What if the "Secret Computer" Crashes?
|
|
|
|
by Sandra Loosemore
March 2, 2003
In a misguided attempt to solve ethical problems with the judging of
figure skating that caused an embarrassment to the sport at the 2002
Olympic Winter Games, this season the ISU has introduced a system of
secret, randomized judging at major competitions. The marks from only a
random subset of the judges -- selected secretly by the scoring computer
-- are used to make the scoring computations, and in addition the
judges' marks are scrambled before they are displayed so that the first
and second marks from individual judges cannot be matched up or
identified. The net effect is that this system makes it impossible for
anyone to independently verify or reconstruct the scoring of skating
competitions.
Aside from the well-known statistical limitations of such a system --
using the marks of all the judges who are present is guaranteed
to produce a more accurate result than using only a subset -- and the
fact that making judging more secret and less accountable can only give
judges who are inclined to cheat more opportunity to do so, this scoring
system has serious practical problems arising from its over-reliance on
computers and computer software. There is no mechanism for
independently verifying that the marks have been entered correctly or
that the software has been programmed to correctly compute the results.
In essence, the ISU's scoring system is an unaudited electronic voting
system, which is a current issue of major concern in the software
engineering community. Many prominent computer
scientists have gone on record as being opposed to such systems; see
this page for more
details.
Even worse, the ISU has proposed a completely new judging system for
figure skating that uses the same "secret and sealed computer" for
random selection of judges' marks, but which is even more dependent on
computer technology and secrecy to implement element-by-element scoring
of a skater's performance. The rule that describes this new judging
system specifies that judges must be equipped with touch-screen displays
and a video replay system, so that they can enter individual scores for
dozens of individual elements within a skater's program (which are to be
identified by a separate expert spotter) in real time, as the skaters
are performing. What's wrong with this picture?
What if the "secret computer" crashes?
This is not just a hypothetical question. There are numerous
documented instances of computer crashes, software bugs or configuration
errors, and operator errors involving the scoring computer at recent
major national and international figure skating competitions. Here are
eight specific instances which I have been able to document through
written reports, television coverage, and eyewitness accounts:
- At the 2002 ISU Grand Prix Final held in December, 2001, the scoring
computer was programmed or configured with the incorrect tie-breaking
rules so that the wrong winners were initially announced in the ice
dance competition. Because the intermediate results had been
displayed, officials who were present were able to immediately
recognize that the final standings produced by the computer were in
error, and take steps to correct the problem.
Source: television coverage. See also:
- At the ISU Junior Grand Prix event held in Scottsdale, Arizona in
September, 2002, the scoring software was somehow misconfigured so
that the marks for required elements and presentation were switched in
the men's short program. This error, which would affect the
tie-breaking procedures, was caught by the referee when the marks for
the first competitor were displayed on the arena scoreboard, and
corrected before the competition continued. The ISU's secret
randomized judging system was not in use at this event; if it had
been, it's possible nobody would have noticed the problem.
Source: eyewitness account by Steven Hazen, sent to me by personal e-mail
- At Skate America in October, 2002, where randomized secret judging
was in use, the web site run by IceCalc (the company which has the
contract for the scoring system software used by the ISU) initially
showed different final results for the compulsory dance competition than
the official standings displayed in the arena; the placements of the 5th
and 6th place couples were reversed, although the marks for both couples
were the same in both versions of the results. The IceCalc web site was
later changed to agree with the official standings, but since no one
knows how the results were computed from the marks in either case, how
can anyone know which set was really correct? Or if either set
was correct?
Source: the original results were copied from the IceCalc web site
and posted on the FSU message board and on the Usenet newsgroup
rec.sport.skating.ice.figure at the conclusion of the event. Follow-up
messages noting the discrepancy with the official results
were also posted there.
- At Skate Canada in November, 2002, where the ISU was using the
secret randomized judging system, the scoring computer apparently
crashed near the beginning of the free dance competition, causing about
a 10-minute delay before the marks for the first couple could be
displayed and the competition could be resumed. It appeared that some
form of manual collection of marks was used until the scoring computer
came back on-line.
Before the adoption of the "secret computer" system, the ISU regulations
specified that there be a second backup computer into which the
accountants manually enter the marks as they are displayed in the arena.
In the event of a failure of the primary computer or scoreboard, it was
common practice for judges to hold up cards with their marks. However,
with the adoption of the randomized secret judging system, these backups
have been removed. Moreover, in the ISU's proposed new judging system
where the judges have to score dozens of elements using a system of
touch-screen menus, what would happen if the computer crashed or an
individual judge's touch-screen device failed in the middle of a
skater's performance?
Source: television coverage; also see eyewitness reports from persons
in attendance:
- At the 2003 US Figure Skating Championships in January, 2003, during
the championship men's free skate, an apparent error by the accountants
operating the scoring computer caused Matt Savoie's marks to be
incorrectly entered under Michael Weiss's name. The competition was
delayed several minutes while the accountants reset the computer and the
judges had to manually re-enter their marks. Because the data used by
the scoring software was left in an inconsistent state, however, it was
not possible to update the standings to show Savoie's placement until
after Weiss had also skated.
Source: the event was broadcast live on ABC; also see:
- At the 2003 European Championships in January, 2003, where the
secret randomized judging system was in use, conflicting sets of results
were again published, this time for the pairs free skate. The incorrect
results appeared on backstage information displays provided for media
and competitors. The Polish skating federation filed a protest on
behalf of their skaters Dorota Zagorska and Mariusz Siudek, who would
have won the bronze medal according to the backstage displays, but who
finished 4th in the official results. The ISU dismissed the complaint,
claiming that the official results had been reviewed and confirmed as
correct. But, because of the randomness and secrecy, it's impossible
for any independent observer to know which set of results -- if either
-- were really correct, same as in the Skate America incident earlier in
the season.
Source:
- At the 2003 Four Continents Championships in February, 2003, where
the ISU's randomized secret judging system was in use, Chinese skater
Dan Fang received this very suspicious set of marks for her free skate:
5.2 5.2 5.2 5.2 5.3 5.3 5.3 5.3 5.3 5.3 5.3 5.3 5.4 5.4
3.5 5.0 5.1 5.1 5.2 5.2 5.2 5.2 5.2 5.2 5.3 5.3 5.4 5.5
Given that Fang had received no other presentation mark lower than a
4.6 in the short program, either, the 3.5 was almost certainly a data
entry error on the part of one of the judges. Before the adoption of
the randomized secret judging system, it was not uncommon for the
referee to confer with judges to verify suspicious-looking marks, and
judges could verify that their own marks were entered correctly when
they were read or displayed on the arena scoreboard. In this case, the
judge who entered the 3.5 may have had no feedback at all to indicate
that he or she had made such a mistake.
Source:
- At the 2003 World Junior Figure Skating Championships in February,
2003, the IceCalc web site showed incorrect warm-up groups for the
ladies free skate while the event was in progress. There is an
exception in the rules for how warm-up groups are decided to allow a
host-country skater to advance to the free skate in addition to the
normal 24 competitors. This skater is added to the first group
so that the groups were 7-6-6-6, but IceCalc's software applied the
default rules and produced the display as if there were 5 groups of 5.
While this didn't affect the computation of the final results in any
way, it's an example of how the scoring software used by the ISU still
contains bugs that tend to crop up embarrassingly in real-life
competition situations.
Source:
Given all of these problems, how can anyone really think that it is a
good idea to trust a "secret and sealed computer", without any mechanism
for verifying either the correctness of the input data or the computed
results?
Recommendations from a software engineer
In order to restore credibility to the way figure skating is scored, the
ISU needs to restore openness and transparency both to the data
and the process used in computing competition results.
Specifically:
- The judges' marks must be displayed in such a way they can be used
for independent computation or verification of the results. It is also
important that individual judges have verification that their intended
marks have been correctly entered into the scoring computer.
- The algorithm used to compute the results must be repeatable and
produce consistent results from the same input data all the time. There
should be no randomness in the results.
- The algorithm used to compute the results should be amenable to
hand computation and not require the use of expensive
computer hardware or proprietary software.
- The rules for figure skating must exist in a formal specification
that can be implemented by anyone, not just encoded in the form of
proprietary software. Merely specifying in the ISU regulations that a
computation or analysis is to be performed "through the computer" is not
an adequate specification of what is to be computed, any more
than requiring the use of an abacus or slide rule would be. The rules
for figure skating should be determined by the ISU according to its
formal, established procedures, rather than by a commercial software
venture which stands to make a considerable profit from licensing its
product to the ISU and the various national skating federations.
- To increase public trust in the correctness of the scoring software,
the ISU should require that only open-source software be certified for
use in competition. This will allow everyone to inspect the software
and verify that it is correctly implementing the rules of the sport. It
will also help in identifying and fixing any bugs in the software
before they crop up in embarassingly public, real-time competition
situations.
Home |
© 1994-2010 SkateWeb |