Increasing universal access to CPDL via MusicXML

Use this forum for HELP at Choral Public Domain Library as well as FEEDBACK
dspreadbury
Posts: 6
Joined: 22 Mar 2015 00:02

Increasing universal access to CPDL via MusicXML

Post by dspreadbury »

I originally posted this on Robert Nottingham's talk page, but he responded that since he is no longer closely involved in the management of CPDL, it would be better to post it here for discussion by the community:

I'm writing you to ask about CPDL's policy on MusicXML files. I believe that contributors to CPDL should be strongly encouraged to make their music available in MusicXML format, for reasons of universal access for blind and visually impaired people, as well as people with other disabilities that affect their ability to read music notation (e.g. dyslexia), who require the use of assistive devices to consume music notation.

Depending on their level of visual impairment, users may require the use of Modified Stave Notation (which you can think of in general terms as "large print music", though there are specific guidelines about how MSN should be prepared, per the UK Association for Accessible Formats), braille music, and talking scores.

Modified Stave Notation is generally tailored for an individual user, since users with partial sight have a wide variety of impairments, and there is no "one size fits all" approach for MSN. Typically MSN is produced by entering the music into one of the existing scoring programs, modified by way of importing a house style or set of libraries that modify many parameters of the music's appearance (stave line thickness, stem thickness, stave size, rhythm dot size, choice of text fonts, choice of paper size, etc. etc.), and then printed out. Due to the unique needs of each user, it is very important that a user should be able to access the music in an interchangeable representation, so that it can be opened in the scoring application available to them, and modified to meet their needs. You can read more about MSN in the UKAAF publication that describes the format as an accessible PDF, here.

Braille music is made using a variety of tools, to lesser or greater degrees of success. The Freedots project is an Open Source MusicXML to braille tool, while the VIP MusicXML to BMML online converter is one outcome of an EU-funded research project and is free to use. Probably the most successful braille music toolset is to open music in Lime and then send it to Goodfeel, a commercial toolset developed by Dancing Dots. If no interchangeable source music is available, then the music must first be input into Lime or another scoring application; given the poor state of accessibility in most scoring software, this generally requires the assistance of a sighted person, and thus immediately reduces access to this music.

Talking scores are useful for people with severe enough visual impairment that MSN is not useful for them, but for whom learning braille music is difficult or impossible, especially for people who lose their sight later in life. No matter your age, learning braille is a daunting challenge. only around half of all blind people can read literary braille, and a smaller proportion still can read the music braille code. UKAAF is working on a new set of guidelines to specify a system for the automatic, computerised transcription of music notation into an accessible verbal description that can be read by assistive technologies such as screen readers and Voice Over on iOS devices. In the past, talking scores have been produced as "scores on tape", transcribed by a sighted person onto an audio cassette. The RNIB, for example, has produced around two dozen talking scores on tape, which are available to borrow from their Music Advisory Service, but the amount of labour required to produce each one is so large that the process of creating them does not scale. Again, a blind person cannot reasonably produce a talking score on his or her own, but if the music is available in an interchangeable format such as MusicXML, then that person can use a software tool to produce a talking score without sighted assistance. You can find some information about the concept behind talking scores on the RNIB's web site

As well as being part of Steinberg's London-based team working on a new commercial scoring application, I am part of the UKAAF's Music Subject Area team, and it is through that role that I am currently focused on the production of guidelines for talking scores, and prototype software that embodies these rules. There are other free software projects ongoing, particularly from the developers of MuseScore (who have beefed up their application's capabilities for producing MSN; are working to make the application itself more accessible to screen readers; are trying to resuscitate the Freedots project and host it online; and are also working on a web service that will allow a PDF to be uploaded to a hosted OMR engine – based on Audiveris – and produce a MusicXML file on demand), that are working to increase universal access to music notation.

The single technology that links all of these efforts is MusicXML. For all of its imperfections, it is the de facto standard for the interchange of symbolic music notation information. Most of the applications used by contributors to CPDL are capable of exporting MusicXML files. Those contributors who already share their binary source files in Sibelius, Finale, or whatever other proprietary formats should be strongly encouraged to create MusicXML versions of those pieces and to upload those as well.

I would also encourage you to include a simple means of searching for scores uploaded in MusicXML format via your advanced search features, to help users in need of accessible formats find them more quickly.

I welcome your feedback, and would love to know what further steps I might take, if any, to push for positive change in further opening up the incredible resource that is CPDL to visually impaired musicians.

With all best wishes,

Daniel Spreadbury (dspreadbury)
CHGiffen
Site Admin
Posts: 1781
Joined: 16 Sep 2005 21:22
Location: Hudson, Wisconsin, USA

Re: Increasing universal access to CPDL via MusicXML

Post by CHGiffen »

As one who has supplied some MusicXML files of works to someone for conversion to Braille, I heartily endorse the notion that MusicXML files of scores are welcome to be published, along with PDF, MP3 (or MIDI), and other source (eg. Finale or Sibelius) files. It would make sense for us to create a MusicXML icon for linking to such files. But, even without the icon, submitted MusicXML files of works would seem to be welcome here.
Charles H. Giffen
CPDL Board of Directors Chair
Admin at & Manager of ChoralWiki
dspreadbury
Posts: 6
Joined: 22 Mar 2015 00:02

Re: Increasing universal access to CPDL via MusicXML

Post by dspreadbury »

I am glad that there is general, non-specific approval of MusicXML files being made available on CPDL, but please allow me to press you again on three specific issues:

1. Can a strong recommendation be made when submitting files in any of the various notation formats (Sibelius, Finale, LilyPond, et al) that a MusicXML file should also be uploaded? (This is actually non-trivial for people using LilyPond, but easily achievable for people using practically any other scoring application.)

2. Can a feature be added to CPDL's search engine that makes it easy to identify works where a MusicXML file is available?

3. For scores that have been uploaded using the CPDL license, is it acceptable for another contributor to convert that edition into MusicXML and upload it back to the site in order to add a MusicXML version without requiring the original contributor to make the new version available?

Thanks for your further consideration.

Daniel
carlos
Site Admin
Posts: 1870
Joined: 19 Aug 2008 15:26
Location: São Paulo, Brasil

Re: Increasing universal access to CPDL via MusicXML

Post by carlos »

Hi Daniel

Replying in reverse order, the answer for number 3 is YES, since derivative works are allowed by the CPDL license.

Number 2 is not something that can be easily implemented, given the current data structure adopted by CPDL. But it's in our plans to have the possibility to also categorize editions by source file format.

As for number 1, I concur with you that recommending the use of a standard format is something that should be reinforced on CPDL. But MusicXML .xml files have the inconvenience of being quite large, due to their textual nature. To avoid this, we currently suggest that such files be uploaded either compressed (inside a .zip, .rar etc.) or using the MusicXML v2.0 .mxl format, that performs an internal compression leading to a file size around one-twentieth the size of the uncompressed version.

Regards,
Carlos (talk)
CPDL Administration
dspreadbury
Posts: 6
Joined: 22 Mar 2015 00:02

Re: Increasing universal access to CPDL via MusicXML

Post by dspreadbury »

Thanks, Carlos.

I am pleased that the CPDL License expressly allows the creation of derivative works in other formats. Can you help me to understand in practical terms what must be done to satisfy the conditions of Section 2 (a) in the CPDL License, specifically concerning the requirement that the derived work should carry "a notice stating that you changed the files and the date of any change"?

By way of example: if I download a CPDL-licensed edition of a work in, say, Sibelius format, which already carries the notice appropriate to satisfy Section 1 of the CPDL License, and then convert that file to MusicXML format and publish it as a new contribution on the site, also under the CPDL License, assuming I have made no other wilful modification to the edition other than converting it from Sibelius format to MusicXML format, what additional notice is required to be given in this case?

Regarding search by format, is it possible that the MediaWiki API supported by the site exposes this information in any way? Are there any limits placed on the use of the API? For example, if it's possible to determine which works have files available in MusicXML format, is there any restriction on using the API to produce a periodically-updated list on an external site of works with MusicXML files, pointing back to the original pages on the CPDL?

Finally, I agree that MusicXML files should ideally be submitted in compressed .mxl format. How would we go about adding recommendations for uploading MusicXML versions in addition to Sibelius, Finale et al editions, and in adding an appropriate icon for MusicXML files for work pages?

Daniel
CHGiffen
Site Admin
Posts: 1781
Joined: 16 Sep 2005 21:22
Location: Hudson, Wisconsin, USA

Re: Increasing universal access to CPDL via MusicXML

Post by CHGiffen »

Finale 2014 has the ability to export to MusicXML, including the 3.0 (.mxl) compressed file format. The compression, is very good. I just exported my 6-part SSATBB Ave verum corpus, and the resulting file size is only 21KB; whereas, the Finale 2014 source file is 147KB, and the PDF sheet music file is 117KB.
Charles H. Giffen
CPDL Board of Directors Chair
Admin at & Manager of ChoralWiki
carlos
Site Admin
Posts: 1870
Joined: 19 Aug 2008 15:26
Location: São Paulo, Brasil

Re: Increasing universal access to CPDL via MusicXML

Post by carlos »

dspreadbury wrote:Can you help me to understand in practical terms what must be done to satisfy the conditions of Section 2 (a) in the CPDL License, specifically concerning the requirement that the derived work should carry "a notice stating that you changed the files and the date of any change"?
I'll ask other admins who are more knowledgeable on this matter to help you with this, but I believe that if you follow the guide below you won't err, because the CPDL license draws from the same principles that originated the CC licenses:

http://mollykleinman.com/2008/08/15/cc- ... nsed-work/
dspreadbury wrote:By way of example: if I download a CPDL-licensed edition of a work in, say, Sibelius format, which already carries the notice appropriate to satisfy Section 1 of the CPDL License, and then convert that file to MusicXML format and publish it as a new contribution on the site, also under the CPDL License, assuming I have made no other willful modification to the edition other than converting it from Sibelius format to MusicXML format, what additional notice is required to be given in this case?
In this case specifically, we'd consider that the MusicXML file is still part of the original edition. A link to it would be added next to the other files of the original editor, and a notice "MusicXML file contributed by Daniel Spreadbury on yyyy-mm-dd." included to the "Edition notes:" section.
dspreadbury wrote:Regarding search by format, is it possible that the MediaWiki API supported by the site exposes this information in any way?
I'm not sure. If you can perform a textual search via the API, then you'll be able to locate the works with available MusicXML files. You could also resort to scanning a dump file of CPDL's database, generated by one of our admins for this purpose.
dspreadbury wrote:Is there any restriction on using the API to produce a periodically-updated list on an external site of works with MusicXML files, pointing back to the original pages on the CPDL?
As far as I know, there are no such restrictions for what you propose.
dspreadbury wrote:How would we go about adding recommendations for uploading MusicXML versions in addition to Sibelius, Finale et al editions, and in adding an appropriate icon for MusicXML files for work pages?
The process for adding a new edition was largely automated last year. The new system was under test for various months, and finally approved. Now the help pages will have to be largely rewritten to reflect the changes, and that could be a good moment to add instructions regarding MusicXML, highlighting the importance of uploading source files in this format.
Carlos (talk)
CPDL Administration
vaarky
Moderator
Posts: 2163
Joined: 22 Aug 2008 07:28

Re: Increasing universal access to CPDL via MusicXML

Post by vaarky »

Previous question:
Is there any restriction on using the API to produce a periodically-updated list on an external site of works with MusicXML files, pointing back to the original pages on the CPDL?

Previous reply:
As far as I know, there are no such restrictions for what you propose.
Is there something we can do to create a page on CPDL where users can go to find a list of scores that are available in MusicXML format on CPDL and provide a pointer to any offsite index that refers to works not indexed on CPDL so people can also find out this type of helpful information when they are on the CPDL site?

Also, did the link Carlos posted cover the question you had about 2(a)? Are you asking about where to put the notice with regard to the MusicXML files, or the wording of the notice? Please clarify any remaining questions.
dspreadbury
Posts: 6
Joined: 22 Mar 2015 00:02

Re: Increasing universal access to CPDL via MusicXML

Post by dspreadbury »

vaarky wrote: Also, did the link Carlos posted cover the question you had about 2(a)? Are you asking about where to put the notice with regard to the MusicXML files, or the wording of the notice? Please clarify any remaining questions.
That link was certainly helpful. I am hoping that, by using the API, we will be able to identify editions that are (a) made available under the CPDL license, and (b) uploaded in a native format (such as Finale, Sibelius, MuseScore), so that we can automatically download the source file, convert it to compressed MusicXML (including adding any required attribution notices), and then upload them to the site. The aim is to dramatically increase the number of MusicXML files available for download from CPDL, to likewise dramatically increase the availability of vocal music that can be converted into accessible formats.

I have taken the liberty of adding a new page in the Help category on the wiki, that I hope the site admins will consider for inclusion in the main Help FAQ, in the "How can I contribute?" section, under "Straightforward questions", preferably immediately after the current question #5 (What should I include in my editions?). The new page is here:

http://www.cpdl.org/wiki/index.php/Help:Format_Choice

The page includes information about MusicXML, the benefits of making music available in an interchange format, and simple instructions for producing compressed MusicXML files in the major scoring applications, plus information about how to compress an uncompressed MusicXML file on Windows and OS X, if compressed MusicXML files cannot be exported by a contributor's scoring program.

I would welcome the use of a dedicated MusicXML icon for editions that have MusicXML files available, and would be glad to provide such an icon myself if that would be helpful, and if you can provide me with the specifications for such a thing.

I would also definitely welcome the creation of a page that lists editions with MusicXML files directly on the site. If this cannot be done, then I would hope it could still be achieved by use of the API and an external page created instead.
vaarky
Moderator
Posts: 2163
Joined: 22 Aug 2008 07:28

Re: Increasing universal access to CPDL via MusicXML

Post by vaarky »

I'll leave the others to reply about some other questions.

Regarding generating a list of songs in MusicXML format, will you index only the ones with CPDL or appropriately derivate-work friendly Licensing type (e.g. certain Creative Commons type licenses supported on the site)? Or all licensing types and, if the latter, then how would you indicate things such as Personal license=contact the contributing editor? It sounds like you were planning to index only CPDL-style licenses but some CC license types would also work.

Thank you for creating that Help page. I made some changes:

I made one change to break a long sentence into two (However, instead of ... but).

Most notably, I changed "Why should I provide my score in MusicXML format" (because it made it sound exclusively MusicXML) to the following:
=Why should I include a MusicXML format version of my score or provide my score in MusicXML format?=

For starters, I modified to add two entries as follows, but others more steeped in user interaction will might have thoughts about the flow (I included both questions because merely having a Format Choice entry will cause people to skip over understanding the benefits of MusicXML if they think merely in terms of what software they use and including a PDF and they know what format they want to use):

http://www.cpdl.org/wiki/index.php/Help:Contents

# [[Help:Style conventions|How do I specify a page or musical work title and other information?]]
# [[Help:Format_Choice|Format Choice]]
# [[Help:Format_Choice#Why_should_I_include_a_MusicXML_format_version_of_my_score_or_provide_my_score_in_MusicXML_format.3F|Why should I include a MusicXML format version of my score or provide my score in MusicXML format?]]
# [[:Category:Music notation programs|Where can I find a list of music notation software?]]
# [[Help:How do I create PDF files?|How do I create PDF files?]]
dspreadbury
Posts: 6
Joined: 22 Mar 2015 00:02

Re: Increasing universal access to CPDL via MusicXML

Post by dspreadbury »

Thanks for making the changes to my Help page. I wonder whether the entry on the main help index should say something like "What format(s) should I use for my edition?" rather than just "Format Choice"? And perhaps I should add some additional context about sharing other formats to the top of the page as well, to make it more generally useful, rather than purely advocating for accessible formats.

Using the wiki markup returned by the API, we have been able to pull together an initial list of works on CPDL, totalling nearly 30,000. What surprised me was that only around 2500 Finale files and around 2300 Sibelius files are available, with practically no MuseScore files at all. Just over 500 editions have MusicXML files available.

This means that even assuming every one of these editions uses a license compatible with the production of derivative works, we would still be able to increase the number of works available in MusicXML format to around 20% of the 19,899 works listed on the home page at the time of writing. This would of course be a big step forwards, but it's a pity that the ceiling will be this low.

After looking at the 'Copyright' values for the editions, my current understanding is that of the dozen or so different licenses that are in use on CPDL, we would be permitted to produce derivative works in MusicXML format for the following license types:
  • CPDL
  • Creative Commons (provided it does not have the "no derivative works" clause, listed separately below)
  • Free Art License
  • GnuGPL
  • Public Domain
However, we would not be permitted to produce derivative works for the following license types:
  • Personal
  • Religious
  • Unknown
  • Creative Commons Attribution No Derivatives
We will do some more work to establish exactly which works lie at the intersection of compatible license types and editions available in symbolic formats that can be converted to MusicXML (Sibelius, Finale, potentially Capella), and then figure out how best to proceed from there.
CHGiffen
Site Admin
Posts: 1781
Joined: 16 Sep 2005 21:22
Location: Hudson, Wisconsin, USA

Re: Increasing universal access to CPDL via MusicXML

Post by CHGiffen »

According to a simple check (using ReplaceText), there are about 4500 pages with Finale 1998-2014 scores available. Moreover, many of these pages have multiple editions in Finale format (eg. works with several movements or editions). However, not all of these are hosted on CPDL servers. The number of works with Finale editions hosted on our own servers seems to be around 2500.
Charles H. Giffen
CPDL Board of Directors Chair
Admin at & Manager of ChoralWiki
vaarky
Moderator
Posts: 2163
Joined: 22 Aug 2008 07:28

Re: Increasing universal access to CPDL via MusicXML

Post by vaarky »

The page on MusicXML format, as it currently reads, is essentially about one factor in considering format choice. So I think "Format Choice" is a closer description than "What format(s) should I use for my edition?" However, the latter is what we should aim for. If you or anyone else can add more context about sharing other formats to that page, that would be very helpful, and I agree that the expanded page should then be retitled on the Help index page. Others here may feel it's ready to be retitled sooner; I defer.

Regarding the Religious category, I am not an attorney, but I wonder if you might index those with the limitation that, per the original contributor's specification, their edition was limited to Religious use and thus any edition you base on it must likewise be limited to Religious use. It may depend on whether you have a good way to present the restriction, versus whether it's simpler to skip those editions. Just an idea...
Claude_T
Moderator
Posts: 609
Joined: 13 Jan 2007 15:16
Location: Fontenay-sous-Bois (France)
Contact:

Re: Increasing universal access to CPDL via MusicXML

Post by Claude_T »

More than one year after:
An icon has been created, thus permitting to generate a count/list of works having at least one MusicXML file here: http://www.cpdl.org/wiki/index.php/User ... ueries/XML.
Thanks to Daniel for pointing us to that direction.
When he wrote his first message, I think we had about 800 works concerned (thanks to Renato Calcaterra), now 1,400.
I exported my 150 editions to MXL. For now, I'm converting those who have PDF and Sibelius (or Finale) only (no MIDI).
Until then, we used MIDI as an interchange format.
I think next step should be converting the PDF-and-MP3-and-Sibelius (or Finale) editions, because MP3 isn't an interchange format.
What do you think?
carlos
Site Admin
Posts: 1870
Joined: 19 Aug 2008 15:26
Location: São Paulo, Brasil

Re: Increasing universal access to CPDL via MusicXML

Post by carlos »

Claude_T wrote:I think next step should be converting the PDF-and-MP3-and-Sibelius (or Finale) editions, because MP3 isn't an interchange format.
What do you think?
Hi Claude, the increasing number of editions with MusicXML source is a good news!

Regarding the MP3 files, I guess that editors upload them in order to guarantee that visitors will have the best listening experience. (MIDI quality depends on the available hardware & software, and can vary a lot.) So, I think we should leave these files alone. As for Sibelius and Finale source files, as far I as know they can be easily exported to MusicXML. I see no problem in doing that and uploading the resulting file to CPDL.
Locked