What if the "Secret Computer" Crashes?

by Sandra Loosemore
March 2, 2003

In a misguided attempt to solve ethical problems with the judging of figure skating that caused an embarrassment to the sport at the 2002 Olympic Winter Games, this season the ISU has introduced a system of secret, randomized judging at major competitions. The marks from only a random subset of the judges -- selected secretly by the scoring computer -- are used to make the scoring computations, and in addition the judges' marks are scrambled before they are displayed so that the first and second marks from individual judges cannot be matched up or identified. The net effect is that this system makes it impossible for anyone to independently verify or reconstruct the scoring of skating competitions.

Aside from the well-known statistical limitations of such a system -- using the marks of all the judges who are present is guaranteed to produce a more accurate result than using only a subset -- and the fact that making judging more secret and less accountable can only give judges who are inclined to cheat more opportunity to do so, this scoring system has serious practical problems arising from its over-reliance on computers and computer software. There is no mechanism for independently verifying that the marks have been entered correctly or that the software has been programmed to correctly compute the results. In essence, the ISU's scoring system is an unaudited electronic voting system, which is a current issue of major concern in the software engineering community. Many prominent computer scientists have gone on record as being opposed to such systems; see this page for more details.

Even worse, the ISU has proposed a completely new judging system for figure skating that uses the same "secret and sealed computer" for random selection of judges' marks, but which is even more dependent on computer technology and secrecy to implement element-by-element scoring of a skater's performance. The rule that describes this new judging system specifies that judges must be equipped with touch-screen displays and a video replay system, so that they can enter individual scores for dozens of individual elements within a skater's program (which are to be identified by a separate expert spotter) in real time, as the skaters are performing. What's wrong with this picture?

What if the "secret computer" crashes?

This is not just a hypothetical question. There are numerous documented instances of computer crashes, software bugs or configuration errors, and operator errors involving the scoring computer at recent major national and international figure skating competitions. Here are eight specific instances which I have been able to document through written reports, television coverage, and eyewitness accounts:

  1. At the 2002 ISU Grand Prix Final held in December, 2001, the scoring computer was programmed or configured with the incorrect tie-breaking rules so that the wrong winners were initially announced in the ice dance competition. Because the intermediate results had been displayed, officials who were present were able to immediately recognize that the final standings produced by the computer were in error, and take steps to correct the problem.

    Source: television coverage. See also:

  2. At the ISU Junior Grand Prix event held in Scottsdale, Arizona in September, 2002, the scoring software was somehow misconfigured so that the marks for required elements and presentation were switched in the men's short program. This error, which would affect the tie-breaking procedures, was caught by the referee when the marks for the first competitor were displayed on the arena scoreboard, and corrected before the competition continued. The ISU's secret randomized judging system was not in use at this event; if it had been, it's possible nobody would have noticed the problem.

    Source: eyewitness account by Steven Hazen, sent to me by personal e-mail

  3. At Skate America in October, 2002, where randomized secret judging was in use, the web site run by IceCalc (the company which has the contract for the scoring system software used by the ISU) initially showed different final results for the compulsory dance competition than the official standings displayed in the arena; the placements of the 5th and 6th place couples were reversed, although the marks for both couples were the same in both versions of the results. The IceCalc web site was later changed to agree with the official standings, but since no one knows how the results were computed from the marks in either case, how can anyone know which set was really correct? Or if either set was correct?

    Source: the original results were copied from the IceCalc web site and posted on the FSU message board and on the Usenet newsgroup rec.sport.skating.ice.figure at the conclusion of the event. Follow-up messages noting the discrepancy with the official results were also posted there.

  4. At Skate Canada in November, 2002, where the ISU was using the secret randomized judging system, the scoring computer apparently crashed near the beginning of the free dance competition, causing about a 10-minute delay before the marks for the first couple could be displayed and the competition could be resumed. It appeared that some form of manual collection of marks was used until the scoring computer came back on-line.

    Before the adoption of the "secret computer" system, the ISU regulations specified that there be a second backup computer into which the accountants manually enter the marks as they are displayed in the arena. In the event of a failure of the primary computer or scoreboard, it was common practice for judges to hold up cards with their marks. However, with the adoption of the randomized secret judging system, these backups have been removed. Moreover, in the ISU's proposed new judging system where the judges have to score dozens of elements using a system of touch-screen menus, what would happen if the computer crashed or an individual judge's touch-screen device failed in the middle of a skater's performance?

    Source: television coverage; also see eyewitness reports from persons in attendance:

  5. At the 2003 US Figure Skating Championships in January, 2003, during the championship men's free skate, an apparent error by the accountants operating the scoring computer caused Matt Savoie's marks to be incorrectly entered under Michael Weiss's name. The competition was delayed several minutes while the accountants reset the computer and the judges had to manually re-enter their marks. Because the data used by the scoring software was left in an inconsistent state, however, it was not possible to update the standings to show Savoie's placement until after Weiss had also skated.

    Source: the event was broadcast live on ABC; also see:

  6. At the 2003 European Championships in January, 2003, where the secret randomized judging system was in use, conflicting sets of results were again published, this time for the pairs free skate. The incorrect results appeared on backstage information displays provided for media and competitors. The Polish skating federation filed a protest on behalf of their skaters Dorota Zagorska and Mariusz Siudek, who would have won the bronze medal according to the backstage displays, but who finished 4th in the official results. The ISU dismissed the complaint, claiming that the official results had been reviewed and confirmed as correct. But, because of the randomness and secrecy, it's impossible for any independent observer to know which set of results -- if either -- were really correct, same as in the Skate America incident earlier in the season.

    Source:

  7. At the 2003 Four Continents Championships in February, 2003, where the ISU's randomized secret judging system was in use, Chinese skater Dan Fang received this very suspicious set of marks for her free skate:

    5.2  5.2  5.2  5.2  5.3  5.3  5.3  5.3  5.3  5.3  5.3  5.3  5.4  5.4
3.5  5.0  5.1  5.1  5.2  5.2  5.2  5.2  5.2  5.2  5.3  5.3  5.4  5.5
    Given that Fang had received no other presentation mark lower than a 4.6 in the short program, either, the 3.5 was almost certainly a data entry error on the part of one of the judges. Before the adoption of the randomized secret judging system, it was not uncommon for the referee to confer with judges to verify suspicious-looking marks, and judges could verify that their own marks were entered correctly when they were read or displayed on the arena scoreboard. In this case, the judge who entered the 3.5 may have had no feedback at all to indicate that he or she had made such a mistake.

    Source:

  8. At the 2003 World Junior Figure Skating Championships in February, 2003, the IceCalc web site showed incorrect warm-up groups for the ladies free skate while the event was in progress. There is an exception in the rules for how warm-up groups are decided to allow a host-country skater to advance to the free skate in addition to the normal 24 competitors. This skater is added to the first group so that the groups were 7-6-6-6, but IceCalc's software applied the default rules and produced the display as if there were 5 groups of 5. While this didn't affect the computation of the final results in any way, it's an example of how the scoring software used by the ISU still contains bugs that tend to crop up embarrassingly in real-life competition situations.
    Source:

Given all of these problems, how can anyone really think that it is a good idea to trust a "secret and sealed computer", without any mechanism for verifying either the correctness of the input data or the computed results?

Recommendations from a software engineer

In order to restore credibility to the way figure skating is scored, the ISU needs to restore openness and transparency both to the data and the process used in computing competition results. Specifically:

SkateWeb
Home © 1994-2010 SkateWeb