Year Book 2019-20 Contents
(Click on underlined link) \/ to end of list \/
Progress with Monthly Grading
The nature of the existing system (“legacy” system) means monthly grading would involve more human intervention and processing than it is reasonable to ask of people, so the intention is to change the system to remedy this.
Present System – In Brief
At the moment, grading calculation takes place in an Access database, which is not publicly accessible on line, and presumably resides in the ECF office. Grading data submission files, containing results of games, have to be e-mailed to a central grading officer who feeds them, if they get through validation, into the Access database.
Currently, that central grading officer processes about 900 submission files per annum, from leagues, congresses and other sources. It is primarily the leagues which would result in more submission files to be processed for a monthly grading system, and they might well increase the number of submission files to (at a wild guess) 2,000 files in total per annum.
Partial Interim Solution
A number of organisers, especially league organisers, enter their results into the ECF League Management System (LMS). Originally, grading submission files were pulled off LMS, and were then checked by an “ECF grader” who would carry out any necessary cleaning up of the data and then e-mail the files to the central grading officer. More recently, in preparation for monthly grading, the facility has been added whereby grading submission data could be extracted “automatically”, by computer, from LMS for submission to the Access database, though bypassing scrutiny by an ECF Grader in this way throws a greater responsibility on the LMS user to ensure data cleanliness, and potentially creates more work centrally in sorting out errors.
This automatic data extraction was tested as part of preparation for the January 2020 grading list, and the testing was “successful”, as somebody once described a computer test run, “in that it highlighted certain errors”! The problems seem to stem mainly from the need for LMS users to be more on the ball. (“Garbage in: garbage out.”) If automatic data extraction is viable then it will significantly help pave the way for monthly grading.
Much grading data is generated outside LMS. This involves processes all of which, one way or another, end up in the production of grading submission files, which get processed by ECF Graders and submitted to the central grading officer by e-mail. Monthly grading would mean that leagues which processed results with their own software (or who even manually prepared data submission files) would place extra work on the plates of their ECF Graders, so making a simpler processing system desirable to handle data from such sources.
Improved Data Submission System
The intent is to create a system whereby files such as those which would hitherto have been e-mailed to the central grading officer can be uploaded by the submitter, to an on-line (MySQL?) database accessed via the web. This receiving database would also do the grading processing, so replacing the present Access database.
The introduction 15 years ago of the on-line database for displaying grades, and for displaying details of the results used to calculate them (slightly misleadingly called “ECF Grading Database”; it does not do the grading), required more information than the legacy system was designed to provide, so certain things started to be placed in a “comments” field to facilitate result documentation. It is intended to incorporate this sort of result documentation in a proper way. Also, there are other things of an administrative nature which might usefully be added into grading data submissions.
It was therefore decided that an enhanced submission file format should be designed for files submitted on line to the above new on‑line database. This format would ultimately replace the current legacy submission formats (of which there are actually two), though use of legacy formats would be catered for during a transition phase, and possibly indefinitely.
Currently, transferring data from the Access grading database to the on-line grade-displaying database is another somewhat manual process, seemingly, and the introduction of an improved data submission file format, with the described extra content formally incorporated, ought to enable this data transfer to be achieved at the click of a button, as it should be.
Progress to Date
Reportedly, “A small team has started building our new grading database as phase 1 of a bigger project. Phase 2 will be looking at accepting results direct from providers with the intention of making all submissions automated with issues highlighted for checking where necessary.”
Also, a proposed new grading data submission file format, as it currently stands during a series of developmental refinements, has now been shared with ECF Graders and others, inviting comment. (This appears to relate to “phase 2” above.)
Some time ago, it was announced that a team of programmers had been assembled, but now it seems some have faded away, and despite the above paragraph’s assertions, there is, I am told, still a need to recruit more help.
Volunteer programmers need, of course, to be web programmers. Writing web-based applications is very different from writing non-web applications. Retired programmers tend not to be web programmers, so web programmers tend to have a “day job”, and so not have enough time to entertain volunteering.
The New Grading Data Submission File Format
This is what is relevant to me as an ECF grader who receives diverse forms of data for conversion to an ECF grading submission file.
Such a format is obviously primarily dictated by the data structure and record content of the database into which the transaction data is being loaded. This structure and content have not been shared fully, so, unless the cart is being put before the horse, it must be that the format is as much as anything serving as the basis of a discussion of what that structure and content should be. It was on that latter basis that I made some initial comments.
A Niggling Worry
Re-reading the file format after looking up the basics of JSON on the web, it rather seemed to me that it had been drawn up by somebody not fully conversant with JSON. There were no square brackets (which enclosed the elements of an array) in sight, and things which would normally form an array were listed as independently named data items. Also, there seemed to be some other misunderstandings of how things work.
I have submitted a rewrite of the JSON file, with corrected syntax etc, though I inadvertently omitted two commas; none of us is perfect.
It is unclear to me therefore how much relevant computing experience and knowledge those involved in this project actually have, and what I have seen inhibits confidence, to be brutally frank.
However, once a new submission file JSON format has been supplied, I am confident adding a routine to my programs to output data in the new format will be relatively simple.
What is Needed
When Malcom Peacock was appointed to produce “LMS”, I experienced an unexpected surge of confidence in the ECF, as I remembered Malcolm, albeit from long ago when he was a chess-playing student at Sheffield University.
This Monthly Grading project really needs another Malcolm Peacock, a single person with the skills and also the time to do the job, or a part of the job.
So, web programmers (with time available), your national chess federation needs you!