Updates to submission process?

Forum for all users to discuss the implementation and operation of the ChoralWiki at CPDL
Richard Mix
Posts: 182
Joined: 31 Jan 2010 04:51

Updates to submission process?

Post by Richard Mix »

I wonder if we could have a News link on the main page: I'm never quite sure whether I'm using the current AddWork form or not: http://www.cpdl.org/wiki/index.php/Form_talk:Add_work

2 unheralded and welcome developments I've noticed just lately are 1) the add edition tab on workpages fills in some work-related blanks in the form automatically, & 2) the file size and number of pages are now filled in when uploading!

2 of my longtime gripes are 1) editor details aren't yet filled in for logged-in users (I leave a filled out add work form in an open tab routinely, but sooner or later there are problems) & 2) it irks me to get an error message because I didn't type my email twice, at the beginning and at the end of the form. Since, as I understand it, the last is provided for non-editor uploaders, shouldn't it be an optional "if different from above" field?
Nikolaus Hold
Posts: 53
Joined: 13 Jul 2016 10:47

Re: Updates to submission process?

Post by Nikolaus Hold »

I confess that I am annoyed every time that I have to type in my e-mail address at all as a registered user, even though it is stored in the user data.
the file size and number of pages are now filled in when uploading!
Could somebody describe in the help pages what I have to do to make this work? I think I use the new add-work-process. I press the upload buttons and get new tabs to upload, but must then enter file size and number of pages in the original tab manually.
choralia
Site Admin
Posts: 2924
Joined: 05 Mar 2006 19:57
Location: Rome, Italy
Contact:

Re: Updates to submission process?

Post by choralia »

The automated add score process was introduced in October 2013. It was announced using the main announcement banner, still available on archive.org (click to enlarge):
archived_announcement.gif
The above announcement was permanently shown for about one year, until October 2014. However, at that time the "Add scores to CPDL" function shown on the navigation panel on the left side of each page was still linking to the "Score submission guide" describing the traditional process, as both processes were (and still are) available to all users. The "Add scores to CPDL" function was eventually re-directed to the page describing the automatic process in November 2016. Contributors gradually migrated from the traditional submission process to the automated process. In 2018, only about 125 submissions out of about 4400 (i.e., approximately 2.8%) have been made using the traditional process.

I have to admit that maintaining both processes available is a little bit cumbersome and sub-optimum, because, to keep the two processes consistent, some limitations of the traditional process tend to propagate to the automatic process. Given the very small number of contributions made with the traditional process nowadays, maybe it's time to remove the traditional process and get rid of its inherent limitations.
Richard Mix wrote:editor details aren't yet filled in for logged-in users
This is intentional because many submissions are made by another user and not by the editor. I'd suggest to activate the browser function that remembers the values entered previously in the same field, so that you get the field completed right away by your browser.
Richard Mix wrote:Since, as I understand it, the last is provided for non-editor uploaders, shouldn't it be an optional "if different from above" field?
Yes, this is possible, although this increases the probability that we don't get the contributor's email address if he/she forgets entering it. This is not a big problem with the automated process as the contributor is anyway identified through the username, while it was a real problem with the traditional process as there were no user identification features, the email address(es) entered were the only ways to identify the contributor or the editor. The automatic process has been designed to mimic the traditional process, so it inherited this specific characteristic.
Nikolaus Hold wrote:I confess that I am annoyed every time that I have to type in my e-mail address at all as a registered user, even though it is stored in the user data.
See above. This is inherited from the traditional process that had no other ways to identify the contributor.
Nikolaus Hold wrote:Could somebody describe in the help pages what I have to do to make this work? I think I use the new add-work-process. I press the upload buttons and get new tabs to upload, but must then enter file size and number of pages in the original tab manually.
Please double check that you upload the pdf file using the "Upload now" button relevant to the box "Sheet music location (give URL, or name of file)". That specific button activates the calculation of the file size and of the number of pages. They are determined after the uploading is successfully completed, so the relevant fields are automatically calculated within few seconds after the uploading is done.
CHGiffen
Site Admin
Posts: 1781
Joined: 16 Sep 2005 21:22
Location: Hudson, Wisconsin, USA

Re: Updates to submission process?

Post by CHGiffen »

I learned awhile ago (after a few misfires) that, with the automated process, one should NOT UPLOAD files BEFORE using the automated process ... but, INSTEAD, upload files using the UPLOAD NOW option within the automated process.

I hope this helps.
Charles H. Giffen
CPDL Board of Directors Chair
Admin at & Manager of ChoralWiki
Nikolaus Hold
Posts: 53
Joined: 13 Jul 2016 10:47

Re: Updates to submission process?

Post by Nikolaus Hold »

I'm sorry, but I seem to be a particularly slow-witted person. I used the "Upload now" buttons every time (there are no others). Last time it worked like this:
  • 1. I clicked on the first/top "Upload now" button and uploaded the pdf file in the new tab. Then I manually took over the URL (right mouse button -> copy link address, then Ctrl-v in the upload tab).
  • 2. Since the page number and size of the PDF file should be automatically filled in, I did not enter it this time.
  • 3. Then I clicked on the 4th/lowest "Upload now" button and uploaded the mxl file in the additional new tab. This time, I did not manually enter the URL in the upload tab to test whether it will be automatically filled in.
  • 4. After filling the rest of the fields, I made the attached hardcopy and then pressed "Add works data".
The result was as follows:

Code: Select all

{{#Legend:}}
*{{PostedDate|2018-12-25}} {{CPDLno|52710}} [[Media:Nh_Schubert_geistlich_Abendlied.pdf|{{pdf}}]] 
{{Editor|NIkolaus Hold|2018-12-25}}{{ScoreInfo|Unknown||}}{{Copy|Creative Commons Attribution Non-Commercial}}
:'''Edition notes:''' Arr. from [[Deutscher Sängerhain (Ernst Schmidt)]]. {{MXL}}
So, no Scoreinfo and no music notation file. What did I do wrong?
Attachments
Bildschirmfoto vom 2018-12-25 11-07-35.png
choralia
Site Admin
Posts: 2924
Joined: 05 Mar 2006 19:57
Location: Rome, Italy
Contact:

Re: Updates to submission process?

Post by choralia »

That's strange. The data of a certain file (e.g., file size and number of a pages in the case of a pdf file) are communicated by the upload form to the add work form through a XML file and AJAX. In the repository of XML files there is no one file associated to your username, so for some reasons the script that determines the file data and writes the XML file is never being activated when you upload a file.

Let's try to narrow down the source of the problem. Could you please run the add work form (you may simulate a new contribution and abort it then, without uploading anything) and check that, when you click the "Upload now" button, the URL of the new tab ends with "Special:Upload?cw_file=score" and your username is shown on the top-right side of the page? The script that writes the XML file only works if "cw_file=score" is present and if you're logged in, so, if one of these conditions is not fulfilled, the XML file is not written and there's no communication from the upload form to the add work form.

Max
Nikolaus Hold
Posts: 53
Joined: 13 Jul 2016 10:47

Re: Updates to submission process?

Post by Nikolaus Hold »

I clicked the "Upload now" button and got the new tab
My user name was shown.

I updloaded the file and entered nothing in my upload tab. Then I got an error, that i did not enter a URL for a media file. So i used again: right mouse button -> copy link address, then Ctrl-v in the upload tab.
Now it worked, bit the result was:

Code: Select all

{{#Legend:}}
*{{PostedDate|2018-12-26}} {{CPDLno|52716}} [[Media:Nh_Silcher_Alle_Jahre_wieder.pdf|{{pdf}}]] 
{{Editor|Nikolaus Hold|2018-12-26}}{{ScoreInfo|Unknown||}}{{Copy|Creative Commons Attribution Non-Commercial}}
:'''Edition notes:''' For {{Vcat|SA}} from [[Kinderlieder in 6 Heften (Friedrich Silcher)]]. 
So, no ScoreInfo. Could it be the first error, or my settings in Firefox, Ubuntu 18.04 (no 3. party cookies, do not create a chronicle)?
choralia
Site Admin
Posts: 2924
Joined: 05 Mar 2006 19:57
Location: Rome, Italy
Contact:

Re: Updates to submission process?

Post by choralia »

This confirms that, for some reasons, the upload form does not communicate with the add work form. It is possible that your specific browser settings interfere with the communication mechanism. Some time ago I struggled with Firefox under Linux due to some strange behavio(u)r, which did not occur at all under Windows. Would it be possible for you to use more permissive settings about cookies under Firefox, or just try using an alternative browser (e.g., Chrome)?

Max
Nikolaus Hold
Posts: 53
Joined: 13 Jul 2016 10:47

Re: Updates to submission process?

Post by Nikolaus Hold »

For my next test, I used a Windows environment:
Win 10, Version 1803,
Google Chrome Version 71.0.3578.98

I uploaded pdf and mxl files with the "Upload now" buttons and initially entered nothing in the upload tab. This led to the known error: the score file name/URL "" appears not to be a valid score file name/URL.

Then I entered both URLs manually - see hardcopy - and clicked on "Add works data".

The result is basically the same (no ScoreInfo):

Code: Select all

{{#Legend:}}
*{{PostedDate|2018-12-28}} {{CPDLno|52732}} [[Media:Nh_Silcher_Vöglein_im_hohen.pdf|{{pdf}}]] [[Media:Nh_Silcher_Vöglein_im_hohen.mxl|{{XML}}]]
{{Editor|Nikolaus Hold|2018-12-28}}{{ScoreInfo|Unknown||}}{{Copy|Creative Commons Attribution Non-Commercial}}
:'''Edition notes:''' From [[Kinderlieder in 6 Heften (Friedrich Silcher)]], Book I, No. 6. {{MXL}}
Attachments
Hardcopy_20181218_cpdl_upl.jpg
choralia
Site Admin
Posts: 2924
Joined: 05 Mar 2006 19:57
Location: Rome, Italy
Contact:

Re: Updates to submission process?

Post by choralia »

Nikolaus Hold wrote:I uploaded pdf and mxl files with the "Upload now" buttons and initially entered nothing in the upload tab. This led to the known error: the score file name/URL "" appears not to be a valid score file name/URL.
The error reported is consistent with a lack of communication between the upload form and the add work form, as no URL is being transferred. I confirm that the XML file that is required for the communication between the two forms is not being written, so we have to understand why this happens.

I'm currently on travel. I'll be back beginning January. Then I'll activate a specific logging and try to identify the reason through that log.

Max
Nikolaus Hold
Posts: 53
Joined: 13 Jul 2016 10:47

Re: Updates to submission process?

Post by Nikolaus Hold »

Let us return to the original issue of Richard Mix and your answer.

I do not mind new links, but the main page should be simplified as a whole because it needs "forever" to load (probably the longer the bigger the CLDP gets).
editor details aren't yet filled in for logged-in users / This is intentional because many submissions are made by another user and not by the editor.
If somebody uploads for another one, he has to fill in the corresponding fields anyway, whether they are filled or not. For them it is not an extra effort, but for the others a relief.
I'd suggest to activate the browser function that remembers the values entered previously in the same field, so that you get the field completed right away by your browser.
This is not an alternative for me because I always work without a browser history for privacy reasons.
no user identification feature / the traditional process that had no other ways to identify the contributor
I do not understand. When I call the form Add_work from the old German instructions for sending in scores, I am first asked to log in. Where is the difference to the new process in the user identification? If there's any loophole about uploading something without logging in with username and password, that's the first thing to do away with.
choralia
Site Admin
Posts: 2924
Joined: 05 Mar 2006 19:57
Location: Rome, Italy
Contact:

Re: Updates to submission process?

Post by choralia »

Nikolaus Hold wrote:the main page should be simplified as a whole because it needs "forever" to load (probably the longer the bigger the CLDP gets).
This not clear to me. Are you referring to the add work form, or to the main page of the CPDL website? The time to render the main page was never mentioned in this topic, so I'm a little bit confused. Anyway, I agree that the main page of the CPDL website includes many complex DPL queries, which make the rendering of the page quite slow. If you login to the main page and display its source code, you should be able to identify the relevant page performance report:

Code: Select all

NewPP limit report
Cached time: 20190101172426
Cache expiry: 0
Dynamic content: true
CPU time usage: 1.104 seconds
Real time usage: 2.412 seconds
Preprocessor visited node count: 5035/1000000
Preprocessor generated node count: 7174/1000000
Post‐expand include size: 31614/2097152 bytes
Template argument size: 8496/2097152 bytes
Highest expansion depth: 16/40
Expensive parser function count: 9/100
Unstrip recursion depth: 0/20
Unstrip post‐expand size: 483/5000000 bytes
If the performance report provided to you is significantly different, I wonder if there is a specific problem with your system, which may be the same problem preventing the correct communication between the upload form and the add work form.
Nikolaus Hold wrote:I do not understand. When I call the form Add_work from the old German instructions for sending in scores, I am first asked to log in. Where is the difference to the new process in the user identification?
The non-automated add work form was originally (year 2005) a simple php script, not integrated within the CPDL wiki and reachable through a URL visible to anybody. The php script could not receive from the wiki any information, such as the username of the contributor, and bots were able to harvest its URL and to make bogus submissions. As a consequence, we received hundreds (or maybe thousands) of bogus submissions made by bots.

Some years later, when I started taking care of the wiki infrastructure, I changed the URL of the script, and embedded the new URL inside a wiki page that is only visible to logged-in users. Now bots, not being logged-in, cannot read the URL and make bogus submissions. This is the reason why you are now requested to login: it's just to prevent that bots can read the URL of the non-automated add work form and make bogus submissions. However, the php script is still not integrated within the wiki, so it is not aware of the contributor's username. Only the automated add work form is integrated within the wiki, and it can retrieve any information available in the wiki database, including the username.

Regarding the required modifications to the way email fields are managed, there are pros and cons in my opinion, so let's see what other admins think about it. For me, no problem to modify the form for the automatic process if there is consensus. A possible consequence may be that the consistency between the automated and the non-automated form cannot be maintained, and the non-automated form should be eventually discontinued.

Max
Nikolaus Hold
Posts: 53
Joined: 13 Jul 2016 10:47

Re: Updates to submission process?

Post by Nikolaus Hold »

The time to render the main page was never mentioned in this topic
Richard Mix wrote: "I wonder if we could have a News link on the main page" and my brain immediately associated, aha, this slow page that takes "forever" to load. I quoted "forever", so hopefully it was clear that it was not meant to be literal. My opinion that the main page is slow, comes mainly from morning breakfast, where I take my tablet, first look at the news and then usually take a look at the CPDL.

I owe it to your interesting reference to the performace report that I have more solid comparative figures available. On my laptop (elderly Core i3, but good enough to make scores with), I took a few samples and looked at the real time usage:

This morning (11:47 - 11:53 German time) the Main_Page needed 1.129 / 0.145 = 7.8 times as long as the page Friedrich_Silcher. However, the page Orlando_di_Lasso even needed 7.99 / 1.129 = 7.1 times as long as the Main_Page. In the afternoon, when the www2 server was unusually slow (16:51-16:57), the following values resulted:
  • Main_Page / Friedrich_Silcher = 11.913 / 0,783 = 15.2
    Main_Page / Help:Contents = 11.913 / 0,102 = 116.8
    Orlando_di_Lasso / Main_Page = 17.195 / 11.913 = 1.4
Of course these are not enough samples, but as a long-term user (with a rather slow internet connection) I can say:
  • The main page often has a response time of >1 second,
    the remaining pages have a response time of <1 second.
The exception, of course, is Orlando_di_Lasso, which is beyond good and evil, but I do not need to look at this page. However, the path to the CPDL usually leads through the main page. That's why I still think the main page should be simplified.
user information, email fields
1. I sympathize with the long-time gripes of Richard Mix, but as you have stated, this must be discussed with the other administrators. I have nothing more to say about that.

2. The automatic transfer of information through the file-upload does not work for me. Since it does not seem to occur to anyone else, it is a singular and therefore completely insignificant occurrence. I mainly wanted to know if this was a user error on my part. I understood that's not the case.

So we could finish this forum topic.
CHGiffen
Site Admin
Posts: 1781
Joined: 16 Sep 2005 21:22
Location: Hudson, Wisconsin, USA

Re: Updates to submission process?

Post by CHGiffen »

As manager of the ChoralWiki and with particular responsibility for the Main Page, I made some changes that will speed up the load time. Mostly, I got rid of some not really necessary DPL queries in Browse ChoralWiki, cut the display in CPDL News down to the current and previous month (instead of the previous 5 months), and I reduced the number of days of Most Recent Scores from 11 to 8 (10 days inclusive to 7 days inclusive).
The performance is indeed better:

Code: Select all

<!-- 
NewPP limit report
Cached time: 20190103010620
Cache expiry: 0
Dynamic content: true
CPU time usage: 0.661 seconds
Real time usage: 1.406 seconds
Preprocessor visited node count: 2355/1000000
Preprocessor generated node count: 6116/1000000
Post‐expand include size: 26288/2097152 bytes
Template argument size: 3876/2097152 bytes
Highest expansion depth: 16/40
Expensive parser function count: 9/100
Unstrip recursion depth: 0/20
Unstrip post‐expand size: 483/5000000 bytes
-->
<!--
Transclusion expansion time report (%,ms,calls,template)
100.00% 1274.673      1 -total
 70.52%  898.923      2 Template:Box
 68.38%  871.575      1 ChoralWiki:TopMatter
 12.77%  162.806      1 ChoralWiki:LatestNews
 12.52%  159.579      2 Template:NewsIn
 11.43%  145.756     62 Template:NewsOn
  6.11%   77.852      1 ChoralWiki:LatestScores
  5.46%   69.603      2 Template:NewWorkCount
  5.24%   66.739      1 ChoralWiki:SearchSection
  4.43%   56.415      1 Template:Bar
-->
If the improvement is significant enough, then we can keep the changes. I can (and will) revert any of the changes if these reductions seem to go too far or don't offer enough improvement on load time for the Main Page.
Charles H. Giffen
CPDL Board of Directors Chair
Admin at & Manager of ChoralWiki
Richard Mix
Posts: 182
Joined: 31 Jan 2010 04:51

Re: Updates to submission process?

Post by Richard Mix »

Since the forum bulletin boards don't show up on my CPDL watchlist I can be slow in acknowledging the thoughtful replies for which Max can be counted on! Still,

1. There are some very confusing redirects about which I asked here:
http://www.cpdl.org/wiki/index.php/Form_talk:Add_work

2. The form asks for: "Name of editor/contributor (First_name Last_name):
Email of editor/contributor:"

and after some scrolling, the bit I always forget: "Your email address (for notification/questions):"

If the editor & contributor are different there's head scratching over which to put on the first line, whereas if everything's adjacent it could just read:
Name of music's editor:
Email of music's editor, if known:
Name of edition's uploader, if different and if Contributor acknowledgement is desired: :P
Email of uploader/form-filler:

3. A minor complaint is the 'cut-along-dotted lines' output of Add works: I often notice Claude doing cleanup like the following http://www.cpdl.org/wiki/index.php?titl ... did=942672. Can the output be in a box that allows command-a or "select all" for copying?
Locked