In fact, the OBO system is not only even more complex than the current ordinal system, but it does not solve the flip-flop problem, either. The OBO system may even introduce flip-flops that would not have occurred under the current system.
This article begins with some background, followed by an overview of how the OBO system works and why it is more computationally complex than the current system. In the third part, some specific examples of flip-flops derived from a simulation of the results of the 1998 World Junior championships are presented. I conclude with some editorializing.
The final results of the men's competition at the 1997 European championships were surprising to many people who did not understand how the ordinal-based scoring system used in figure skating works, or who were not paying close attention to the scoring as the event unfolded. Complicating the situation, the press covering the event (e.g., TV commentators such as Dick Button) seemed to be at a complete loss to explain the scoring system or results to the public. For more details about this, see my previous article at http://www.frogsonice.com/skateweb/obo/score-tech.shtml.
Following the European Championships, Ottavio Cinquanta, president of the International Skating Union, announced that he wanted a new scoring system adopted. Cinquanta has made repeated statements to the press that the new scoring system must not permit "flip-flops" in the standings, where the relative placements of two competitors who have already skated are changed by the marks given to a competitor who skates later, such as what happened at 1997 Europeans. (Most recently, at the 1997-98 Champions Series final in Munich, Cinquanta was quoted in press reports as stating that "If one skater is in front of another, he should remain there.") Also getting into the act was IOC chair Juan Antonio Samaranch, who made public statements to the effect that the system used to score figure skating at the Olympics must be understandable to the public.
In fact, the ISU's own regulations did not allow changes to the scoring system to be adopted in time for the 1998 Olympic Winter Games. In the meantime, while various other alternatives were presented, Mr. Cinquanta focused on the "OBO" proposal for a new scoring system presented to the ISU by Andreas Sigurdsson and sponsored by the DEU (the German federation). The ISU technical committees were instructed to allow the DEU to test the OBO system at the Nebelhorn Trophy competition in August, 1997. The results of that test were inconclusive and subject to dispute as to what a "change of position" really was. Rumors then began swirling at the 1998 World Junior championships that the marks awarded there would be recalculated under OBO and presented to the technical committees for further review during meetings to be held at the Champions Series final.
|OBO: What it is, how it works|
OBO stands for "one-by-one". The general idea behind this scoring system is to compare the skaters pairwise and rank them according to which has the most total "wins" in these pairwise comparisons. Some people may find the idea of ranking skaters according to a "who-beat-who" criteria appealing, but this is definitely not how the OBO system works. It is quite possible under OBO for skater X to place ahead of skater Y even if Y beat X in a head-to-head comparison of marks, for example.
The OBO system does not introduce any changes to the marking used by the judges. Under OBO it is also convenient to compute ordinals in the same way as for the current scoring system since this simplifies the later computations somewhat. However, the manner of using the ordinals from the different judges to calculate standings is completely different than for the current system. Although statements in the press suggest this is just a minor change to the current system to prevent flip-flops, the actual differences in standings between the two systems can be dramatic.
If there are N skaters participating in the event, the OBO system requires building an N-by-N table, with a row and column for each skater. The entries in the table are filled in by comparing each skater against all the others. In comparing skater X against skater Y, there are two pieces of information that have to be filled in each table entry: whether X skater has "won" a majority of judges when compared to Y, and the number of judges that voted for X over Y (this is called "Judges in Favor", or "JiF"). Then the total number of "wins" for each skater is added up, as is the total number of "Judges in Favor". The skaters are ranked by total wins with total JiF used as the tie-breaker.
Here's an example involving a small event with only 6 competitors (derived from the actual marks given to the top 6 finishers in the free skating in the men's event at 1997 Europeans). The first step is computing the ordinals in the usual way. Suppose this works out to give us:
A 1 1 1 1 1 2 1 1 1 B 3 2 5 2 3 3 5 6 6 C 5 5 4 4 2 4 2 2 3 D 4 3 3 6 4 6 4 3 2 E 2 4 2 3 6 5 3 4 5 F 6 6 6 5 5 1 6 5 4The OBO comparison table might look like this:
|| A | B | C | D | E | F || total wins || total JiF ----++-----+-----+-----+-----+-----+-----++--------------++------------- A || | 1/9 | 1/9 | 1/9 | 1/9 | 1/8 || 5 || 44 ----++-----+-----+-----+-----+-----+-----++--------------++------------- B || 0/0 | | 0/4 | 1/5 | 0/4 | 1/6 || 2 || 19 ----++-----+-----+-----+-----+-----+-----++--------------++------------- C || 0/0 | 1/5 | | 1/5 | 1/5 | 1/8 || 4 || 23 ----++-----+-----+-----+-----+-----+-----++--------------++------------- D || 0/0 | 0/4 | 0/4 | | 0/4 | 1/7 || 1 || 19 ----++-----+-----+-----+-----+-----+-----++--------------++------------- E || 0/0 | 1/5 | 0/4 | 1/5 | | 1/6 || 3 || 20 ----++-----+-----+-----+-----+-----+-----++--------------++------------- F || 0/1 | 0/3 | 0/1 | 0/2 | 0/3 | || 0 || 10 ----++-----+-----+-----+-----+-----+-----++--------------++-------------So in this case the results would be A, C, E, B, D, F. (Under the current scoring system, the results were A, B, C, D, E, F; so you can see that the OBO system does indeed produce results that are quite different.)
Aside from any questions regarding the effectiveness of the OBO scoring system, such as whether or not it is resistant to judging errors or prevents flip-flops in the standings during the course of an event, it is immediately obvious that it is much more complex from a computational point of view than the current system. The current system requires only a linear scan of the ordinals given to each competitor to compute the ranking criteria (the majority, total ordinals of majority, and so on). The OBO system, on the other hand, requires an order N-squared comparison followed by a linear scan.
The additional computational complexity may not appear to be a big issue in the age of ubiquitous computers, but in fact many skating fans who attend competitions can and do work out the results by hand using the scoring sheets provided in the program booklet. At competitions that do not have electronic displays of running standings, fans often work out the results by hand long before the official results are posted. Under the OBO system, doing this would at best be very impractical. Certainly, at an event like the World Championships when there are up to 30 competitors, a very large sheet of paper would be necessary to work out or display the 30-by-30 table of results, and it would be extremely tedious and error-prone to have to do at least an additional 3,915 comparisons by hand, along with repeated additions of 30 columns of numbers, to fill in the table.
A further problem regarding the complexity of the OBO system is one of the issues that initially led to the search for a new scoring system: the complaint that the scoring system was too complex for the TV-viewing public to understand or for the media to explain. Introducing a scoring system that is even more complex, and that in many instances produces results that are just as counter-intuitive, is certainly not going to solve this problem.
|So what about flip-flops?|
In fact, the OBO system does not prevent flip-flops in the event standings -- instances where the relative placements of two skaters change because of the placements given to a third skater by the judges.
To show how the OBO system might work in a real-life competition situation, a simulation was performed using the marks and skate orders from the 1998 World Junior competition, available on the web at http://www.wige.de/Skating/cgi-bin/subev.idc?eevt_id=9. (Recall that the OBO system doesn't change the interpretation of the judges' marks from the current rules, only the manner in which the results are computed from the marks.) This simulation shows that, under the OBO system, flip-flops would have occurred in at least four phases of the competition.
It is also interesting to observe that Nikodinov actually scored a head-to-head "win" over Fivian, and Sebestyen "won" over Nikodinov -- exactly the opposite of their final placements in the OBO simulation.
In this event as well, there were instances where the overall placements of the competitors concerned did not reflect the "wins" in head-to-head comparisons: e.g., Vlandis & Guzman actually "won" over Lefrancois & Osseland.
The fun started when Alexei Kozlov skated. Before this, the standings for the top 10 were:
Wins 1 Delmore 15 2 Davydov 14 3 Danilchenko 13 4 Hiraike 12 5 Takeuchi 11 6 Sandhu 10 7 Ferriera 9 8 Davies 8 9 Street 7 10 Murvanidze 6Kozlov was given a mixed bag of marks that gave him "wins" over Ferriera and Davies but not Street or Murvanidze. This meant that Davies, Street, and Kozlov now all had 8 "wins", and moreover, Davies and Street were in an absolute tie with the same total "Judges in Favor". Note that at this point Kozlov is behind two of the skaters he beat in head-to-head competition, and ahead of one who beat him!
Wins JiF 1 Delmore 16 2 Davydov 15 3 Danilchenko 14 4 Hiraike 13 5 Takeuchi 12 6 Sandhu 11 7 Ferriera 9 8 Davies 8 72 8 Street 8 72 10 Kozlov 8 65 11 Murvanidze 7These relative standings remained unchanged after the next two skaters, Gruber and Amentas, who finished farther down the list. Next up was Lee, who received marks that gave him "wins" over Sandhu and all the skaters below him. However, the individual judges' placements of Lee compared to Davies and Street broke the tie on "Judges in Favor" between them: judge #5 voted for Street over Lee, while all the judges had Davies behind Lee. So now, the placements of Davies and Street flip-flopped:
Wins JiF 1 Delmore 19 2 Davydov 18 3 Danilchenko 17 4 Hiraike 16 5 Takeuchi 15 6 Lee 14 7 Sandhu 13 8 Ferriera 11 9 Street 10 89 10 Davies 10 88 11 Kozlov 10 81 12 Murvanidze 9The next skater was Sviatko. His marks placed him even farther up the standings than Lee, but this time Davies picked up one judge on him, thanks to his generous marks from judge #6. So, Davies and Street were back in an absolute tie!
Wins JiF 1 Delmore 20 2 Davydov 19 3 Danilchenko 18 4 Hiraike 17 5 Sviatko 16 6 Takeuchi 15 7 Lee 14 8 Sandhu 13 9 Ferriera 11 10 Street 10 89 10 Davies 10 89 12 Kozlov 10 81 13 Murvanidze 9The next skater after this was Rakowski. Rakowski's marks were also high enough to place him well above the skaters involved in the tie, but once again Davies' high marks from judge #6 were enough to give him one more vote and break the tie with Street yet again, this time in his favor rather than Street's. Davies then retained this lead for the remainder of the competition.
Rakowski's placement is interesting in another way: judge #4 gave him the identical marks as Sviatko, and since the other 8 judges were split evenly, this also caused an absolute tie in determining the "win" in the head-to-head comparison between them. This in turn led to a tie in the number of "Wins", which was broken by "Judges in Favor".
Wins JiF 1 Delmore 21 2 Davydov 20 3 Danilchenko 19 4 Hiraike 18 5 Sviatko 17 142 6 Rakowski 17 139 7 Takeuchi 15 8 Lee 14 9 Sandhu 13 10 Ferriera 11 11 Davies 10 90 12 Street 10 89 13 Kozlov 10 81 14 Murvanidze 9
First of all, it is very obviously the case that the OBO system does not eliminate flip-flopping within the standings. It is possible that it may even introduce more flip-flops than the current scoring system, although more statistical analysis of the two systems would have to be performed to determine this.
In situations where there is no clear consensus among the judges about the placements of skaters, the simulation shows it is quite common for the eventual results of the OBO system to place a skater behind another competitor in spite of "winning" a head-to-head confrontation with that competitor. The men's short program scenario above even included an instance where a skater "won" against the competitors who were two and three places above him. This is troubling even in instances where it does not actually cause a flip-flop in the standings, because it indicates situations where additional flip-flops could have occurred if a different skating order had been drawn. It also means that OBO as a system is not internally consistent as to the significance of one-on-one "wins".
Another disturbing problem with the OBO system lies with using "Judges in Favor" as the sole tie-breaking mechanism. The problem is that a single judge who gives an ordinal that is greatly out-of-line with rest of the panel can wind up having a disproportionate effect should skaters wind up tied on the number of "wins". And ties on "wins" seem to be quite common in cases where the judging panel is split between a group of three or more skaters.
It is not clear to me why anyone connected with the ISU is spending any more time on evaluating or advocating the OBO system, since it meets neither of the supposed criteria for a new scoring system -- being easy to understand, and eliminating flip-flops. It is also inferior to the current system in that judging errors are more likely to affect the overall standings. It certainly seems better to leave well enough alone and concentrate on educating the public and the media about the current scoring system or seeking out improved ways to present the results so that they are more understandable to the public.
It is also not clear to me why Ottavio Cinquanta released a statement to the press during the Champions Series Final in Munich, announcing the adoption of the OBO system as the solution to the "flip-flop" problem before the technical committee meeting during the event had even finishished preparing its report. If in fact the ISU's technical committees did use 1998 World Juniors as a test case, they must have become aware of the same problems and had much the same conclusions about them. From the press reports, it appears that Cinquanta also asserted that the OBO proposal had the backing of the ISU council, when in fact the proposal apparently has only a minority of clear supporters among the ISU council and technical committee members. By making premature and misleading claims on behalf of the OBO system, it seems like Cinquanta has merely created more confusion about how skating is scored.
|Home||© 1994-2010 SkateWeb|