OBO: A Detailed Analysis

OBO: A Detailed Analysis

By Sandra Loosemore (March, 1998)

The ISU is considering adoption of a new scoring system referred to as the "OBO" system, as a result of complaints that the current ordinal scoring system is too complicated and sometimes leads to "flip-flops" in the event standings -- situations where the relative rankings of two competitors change as a result of the marks given to a third competitor who skates later.

In fact, the OBO system is not only even more complex than the current ordinal system, but it does not solve the flip-flop problem, either. The OBO system may even introduce flip-flops that would not have occurred under the current system.

This article begins with some background and a summary of the ISU's actions concerning the OBO proposal. This is followed by an overview of how the OBO system works and why it is more computationally complex than the current system. In the third part, some specific examples of flip-flops derived from simulations of the results of the 1998 World Junior Championships, 1998 European Championships, and 1998 Olympic Games are presented. I conclude with some editorializing.

Background

In the men's free skate at the 1997 European Championships, there was a dramatic reshuffling of placements owing to the fact that, in a very close competition, the marks given to the last skater caused one of the previous competitors to lose a majority for the place he had temporarily been holding. This "flip-flop" in the standings was surprising to many people who did not understand how the ordinal-based scoring system used in figure skating works, or who were not paying close attention to the scoring as the event unfolded. Complicating the situation, the press covering the event seemed to be at a complete loss to explain the scoring system or results to the public.

Following the event, Ottavio Cinquanta, president of the International Skating Union, stated that he thought the current scoring system was unsatisfactory and should be changed. Cinquanta has made repeated statements to the press that the scoring system must not permit "flip-flops" in the standings, where the relative placements of two competitors who have already skated are changed by the marks given to a competitor who skates later, such as what happened at 1997 Europeans. (Most recently, at the 1997-98 Champions Series Final in Munich, Cinquanta was quoted in press reports as stating unequivocally that "If one skater is in front of another, he should remain there.") Also getting into the act was IOC chair Juan Antonio Samaranch, who has made public statements to the effect that the system used to score figure skating at the Olympics must be understandable to the public.

In fact, the ISU's own regulations did not allow changes to the scoring system to be adopted in time for the 1998 Olympic Winter Games. In the meantime, while various other alternatives were presented, Cinquanta focused on the "OBO" proposal for a new scoring system presented to the ISU by Andreas Sigurdsson and sponsored by the DEU (the German federation).

The ISU technical committees were instructed to allow the DEU to test the OBO system at the Nebelhorn Trophy competition in August, 1997. The results of that test were inconclusive and subject to dispute as to what a "change of position" really was. A second test was later commissioned in the form of a simulation: the results of the 1998 World Junior championships were to be recalculated under OBO during meetings of the technical committees at the Champions Series Final in Munich in December, 1997. However, before the technical committees could submit their report, Cinquanta announced to the press that OBO had been approved!

Meanwhile, at the Munich meeting, Cinquanta claimed that the provision in the ISU Constitution permitting the Council to assign tasks to the technical committees and another provision to the effect that the Vice-President for Figure Skating "oversees" the work of the figure skating technical committees in essence gave the President the authority to "direct" the outcome of work by the committees. On that basis, the committees were instructed to approve OBO unless it had a "major flaw". They were further informed that problems identified in the simulation and in prior examination of OBO (including flip-flops, significantly different outcomes, problems with manual calculation, high costs of software for the new system, etc.) did not constitute "major flaws". Thus, any further technical evaluation of the OBO proposal by the ISU was effectively squelched and to a large extent the only "evaluation" done was strictly limited to what Cinquanta permitted the technical committees to do.

Even more incredibly, Cinquanta then issued a statement to the ISU membership (Circular Letter No. 536) claiming that the evaluation based on the World Junior Champions had "officially been made", and further implying that the technical committees had endorsed the OBO proposal as a result of the test. The truth of the matter is that the results of the second evaluation were simply discarded and that the proposal was "accepted" by the committees only as a result of political maneuvering, over any and all technical objections.

As a result of these actions, the OBO proposal is now on the agenda (as items 414, 415, 416, and 417) for consideration at the ISU Congress in June 1998. It must be approved by a 2/3 vote of ISU member organizations in order to take effect.

OBO: What it is, how it works

OBO stands for "one-by-one". The general idea behind this scoring system is to compare the skaters pairwise and rank them according to which has the most total "wins" in these pairwise comparisons. Some people may find the idea of ranking skaters according to a "who-beat-who" criteria appealing, but this is definitely not how the OBO system works. It is quite possible under OBO for skater X to place ahead of skater Y even if Y beat X in a head-to-head comparison of marks, for example.

The OBO system does not introduce any changes to the marking used by the judges. Under OBO it is also convenient to compute ordinals in the same way as for the current scoring system since this simplifies the later computations somewhat. However, the manner of using the ordinals from the different judges to calculate standings is completely different than for the current system. Although statements from Cinquanta and others that have been reported in the press suggest OBO is just a minor change to the method of computing running standings to prevent flip-flops, the two systems can produce dramatic differences in the final standings.

If there are N skaters participating in the event, the OBO system requires building an N-by-N table, with a row and column for each skater. The entries in the table are filled in by comparing each skater against all the others. In comparing skater X against skater Y, there are two pieces of information that have to be filled in each table entry: whether X skater has "won" a majority of judges when compared to Y, and the number of judges that voted for X over Y (this is called "judges in favor", or "JiF"). Then the total number of "wins" for each skater is added up, as is the total number of "judges in favor". The skaters are ranked by total wins with total JiF used as the tie-breaker.

Here's an example involving a small event with only 6 competitors (derived from the actual marks given to the top 6 finishers in the free skating in the men's event at 1997 Europeans). The first step is computing the ordinals in the usual way. Suppose this works out to give us:

A   1  1  1  1  1  2  1  1  1
B   3  2  5  2  3  3  5  6  6
C   5  5  4  4  2  4  2  2  3
D   4  3  3  6  4  6  4  3  2
E   2  4  2  3  6  5  3  4  5
F   6  6  6  5  5  1  6  5  4

The OBO comparison table might look like this:

    ||  A  |  B  |  C  |  D  |  E  |  F  ||  total wins  ||  total JiF
----++-----+-----+-----+-----+-----+-----++--------------++-------------
 A  ||     | 1/9 | 1/9 | 1/9 | 1/9 | 1/8 ||     5        ||    44
----++-----+-----+-----+-----+-----+-----++--------------++-------------
 B  || 0/0 |     | 0/4 | 1/5 | 0/4 | 1/6 ||     2        ||    19
----++-----+-----+-----+-----+-----+-----++--------------++-------------
 C  || 0/0 | 1/5 |     | 1/5 | 1/5 | 1/8 ||     4        ||    23
----++-----+-----+-----+-----+-----+-----++--------------++-------------
 D  || 0/0 | 0/4 | 0/4 |     | 0/4 | 1/7 ||     1        ||    19
----++-----+-----+-----+-----+-----+-----++--------------++-------------
 E  || 0/0 | 1/5 | 0/4 | 1/5 |     | 1/6 ||     3        ||    20
----++-----+-----+-----+-----+-----+-----++--------------++-------------
 F  || 0/1 | 0/3 | 0/1 | 0/2 | 0/3 |     ||     0        ||    10
----++-----+-----+-----+-----+-----+-----++--------------++-------------

So in this case the results would be A, C, E, B, D, F. (Under the current scoring system, the results were A, B, C, D, E, F; so you can see that the OBO system does indeed produce results that are quite different.)

Aside from any questions regarding the effectiveness of the OBO scoring system, such as whether or not it is resistant to judging errors or prevents flip-flops in the standings during the course of an event, it is immediately obvious that it is much more complex from a computational point of view than the current system. The current system requires only a linear scan of the ordinals given to each competitor to compute the ranking criteria (the majority, total ordinals of majority, and so on). The OBO system, on the other hand, requires an order N-squared comparison followed by a linear scan.

The additional computational complexity may not appear to be a big issue in the age of ubiquitous computers, but in fact many skating fans who attend competitions can and do work out the results by hand using the scoring sheets provided in the program booklet. At competitions that do not have electronic displays of running standings, fans often work out the results by hand long before the official results are posted. Under the OBO system, doing this would at best be very impractical. Certainly, at an event like the World Championships when there are up to 30 competitors, a very large sheet of paper would be necessary to work out or display the 30-by-30 table of results, and it would be extremely tedious and error-prone to have to do at least an additional 3,915 comparisons by hand, along with repeated additions of 30 columns of numbers, to fill in the table.

A further problem regarding the complexity of the OBO system is one of the issues that initially led to the search for a new scoring system: the complaint that the scoring system was too complex for the TV-viewing public to understand or for the media to explain. Introducing a scoring system that is even more complex, and that in many instances produces results that are just as counter-intuitive, is certainly not going to solve this problem.

So what about flip-flops?

In fact, the OBO system does not prevent flip-flops in the event standings -- instances where the relative placements of two skaters change because of the placements given to a third skater by the judges.

To show how the OBO system might work in a real-life competition situation, simulations were performed (independently of the tests commissioned by the ISU) using the marks and skate orders from the 1998 World Junior and 1998 European championships, available on the web at http://www.wige.de/Skating/index.htm, and similar information from the 1998 Olympic Games, available at http://www.nagano.olympic.org/. (Recall that the OBO system doesn't change the interpretation of the judges' marks from the current rules, only the manner in which the results are computed from the marks.) These simulations showed that, under the OBO system, flip-flops would have occurred in seven of the fourteen competition segments at World Juniors; three of ten competition segments at Europeans; and four of ten competition segments at the Olympic Games. Specifically:

At the World Junior Championships:

Ladies Qualifying Round B, involving Angela Nikodinov, Anina Fivian, and Sara Lindroos;
Pairs Short Program, involving Natalie Vlandis & Jered Guzman and Sabrina Lefrancois & Nicolas Osseland;
Compulsory Dance 2, involving Natalia Romaniuta & Danil Barantsev and Jamie Silverstein & Justin Pekarek;
Original Dance, again involving Natalia Romaniuta & Danil Barantsev and Jamie Silverstein & Justin Pekarek;
Men's Short Program, involving Matthew Davies, Alan Street, and Alexei Kozolov;
Men's Free Skate, involving Emanuel Sandhu and Ben Ferriera.
Ladies Free Skate, involving Huan Wang and Yea-Ji Shin.

At the European Championships:

Ladies Short Program, involving Jekaterina Golovatenko, Anina Fivian, and Katerina Blohonova;
Ladies Free Skate, involving Krisztina Czako, Julia Lautova, and Yulia Vorobieva;
Men's Free Skate, involving Patrick Meier and Ivan Dinev.

At the Olympic Games:

Men's Short Program, three separate instances: one involving Michael Weiss, Michael Shmerkin, and Szabolcs Vidrai; the second involving Igor Pashkevitch and Yamato Tamura; and the third involving Margus Hernits, Cornel Gheorghe, and Roman Skorniakov.
Men's Free Skate, involving Jeff Langdon and Michael Tyllesen.
Compulsory Dance 1, involving Irina Romanova & Igor Yaroshenko and Margarita Drobiazko & Povilas Vanagas.
Ladies Free Skate, involving Anna Rechnio and Alisa Drei.

To summarize the sixteen flip-flop situations that were found in these OBO simulations:

Eight of the cases involved a 3-way tie on "wins": skater A "wins" over B, B "wins" over C, and C "wins" over A. There is no way that A, B, and C can be ranked without going contrary to one of the "wins" and possibly reversing their initial placements, depending on the skate order. Moreover, because of fluctuations in the "judges in favor" for each skater, the tie may be broken in different ways at different points during the event, causing multiple flip-flops between the same competitors. (There were also at least three other instances of a 3-way tie on "wins" that did not cause flip-flops simply because of the skate order.)
In seven cases the flip-flop arose out of a similar near-tie situation involving 4 or more competitors who could not be ranked without reversing at least one "win". Two of these flip-flop instances were within the same group of 6 competitors whose "wins" against each other formed convoluted circular relationships.
One case involved a tie on "wins" between two skaters only, as a result of a judge giving both skaters identical marks while the rest of the panel was evenly split between them. (There was at least one other other instance of this happening that did not result in a flip-flop.)

We will now examine each of these instances in more detail.

World Juniors, Ladies Qualifying Round B

In the OBO simulation, Angela Nikodinov was initially ahead of Anina Fivian on "wins", but the marks for Sara Lindroos gave all three skaters the same number of "wins". Breaking the tie by "judges in favor" put Nikodinov behind both Fivian and Lindroos. Later, Nikodinov moved into an absolute tie with Lindroos, and then dropped again behind her again, because of changes in the "judges in favor".

It is interesting to observe that Nikodinov actually scored a head-to-head "win" over Fivian, who finished two places ahead of her in the simulation.

World Juniors, Pairs Short Program

In the OBO simulation, after 9 teams had skated, Natalie Vlandis & Jered Guzman and Sabrina Lefrancois & Nicolas Osseland were in second and third places, respectively, in a tie by total "wins" but with V&G slightly ahead by "judges in favor". However, after Jacinthe Lavriere & Lenny Faustino skated, the placements of V&G and L&O flip-flopped because L&O received more "judges in favor" against L&F than did V&G. This happened even though L&F placed well below (5th) both of the pairs who were affected by the flip-flop, and is an example of how erratic marks given by a minority of judges can have a major impact on the overall placements of the competitors under the OBO system.

In this event as well, there were instances where the overall placements of the competitors concerned did not reflect the "wins" in head-to-head comparisons: e.g., Vlandis & Guzman actually "won" over Lefrancois & Osseland, and at one point in the competition were ahead of them by total "wins".

World Juniors, Compulsory Dance 2

With one team yet to skate, the standings showed these three teams nicely ordered:

                                    Wins JIF
 6.             HRAZSKA / PROCHAZKA  16  138
 7.           ROMANIUTA / BARANTSEV  15  132
 8.           SILVERSTEIN / PEKAREK  14  130

Kristina Kobaladze & Oleg Voiko's marks created a very interesting situation: they scored "wins" over Gabriela Hrazska & Jiri Prochazka and Natalia Romaniuta & Danil Barantsev, but not over Jamie Silverstein & Justin Pekarek. This created two ties on "wins" and caused S&P to move ahead of R&B on "judges in favor":

                                    Wins JIF
 6.             HRAZSKA / PROCHAZKA  16  142
 7.               KOBALADZE / VOIKO  16  141  <=== new
 8.           SILVERSTEIN / PEKAREK  15  135  <=== changed
 9.           ROMANIUTA / BARANTSEV  15  134  <=== changed

World Juniors, Original Dance

Here there was another flip-flop involving Jamie Silverstein & Justin Pekarek and Natalia Romaniuta & Danil Barantsev, this time after Aleksandra Kauc & Filip Bernadowski skated. At one point, the two teams were tied on "wins" but where Silverstein & Pekarek had a greater number of "judges in favor". When K&B skated, they "won" against S&P but not R&B, which broke the tie in "wins" between the latter two teams. However, even though R&B wound up with more "wins", as the placements for the remaining skaters in the competition were computed, R&B continued to have fewer "judges in favor" than S&P.

Closer examination of this flip-flop situation reveals that there were actually five couples involved in a near-tie situation that could not be resolved without reversing "wins": K&B and R&B both also scored "wins" over Kristina Kobaladze & Oleg Voiko, who "won" over Gabriela Hrazska & Jiri Prochazka, who in turn "won" over the three other couples involved. Again in this competition segment, it was observed that a single judge who gave marks that were out of line could have a significant effect in the overall standings due to the "judges in favor" tie-breaker. For example, S&P's higher "judges in favor" count could be attributed to the scattering of 5th, 6th, and 7th place votes they received although they had a majority only for 10th place. Meanwhile, R&B's one 11th-place vote among their majority for 8th place essentially meant that the one judge cost them three "judges in favor" points.

World Juniors, Men's Short Program

This was probably the most interesting and complex of the flip-flops at World Juniors, because it involved numerous skaters, two flip-flops, absolute ties, and placements being changed as a result of marks given to skaters who placed nowhere near the skaters affected. It is probably easiest to present this scenario by showing the standings of the skaters as the event unfolded in the simulation.

The fun started when Alexei Kozlov skated. Before this, the standings for the top 10 were:

                                    Wins JIF
 1.                 Derrick DELMORE  15  132
 2.                  Sergei DAVYDOV  14  122
 3.              Vitaly DANILCHENKO  13  123
 4.                  Taijin HIRAIKE  12  108
 5.                 Yosuke TAKEUCHI  11   96
 6.                  Emanuel SANDHU  10   87
 7.                    Ben FERREIRA   9   72
 8.                  Matthew DAVIES   8   68
 9.                     Alan STREET   7   66
10.             Vakhtang MURVANIDZE   6   54

Kozlov was given a mixed bag of marks that gave him "wins" over Ferriera and Davies but not Street or Murvanidze. This meant that Davies, Street, and Kozlov now all had 8 "wins", and moreover, Davies and Street were in an absolute tie with the same total "judges in favor". Note that at this point Kozlov was behind two of the skaters he beat in head-to-head competition, and ahead of one who beat him!

                                    Wins JIF
 1.                 Derrick DELMORE  16  141
 2.                  Sergei DAVYDOV  15  131
 3.              Vitaly DANILCHENKO  14  132
 4.                  Taijin HIRAIKE  13  117
 5.                 Yosuke TAKEUCHI  12  105
 6.                  Emanuel SANDHU  11   95
 7.                    Ben FERREIRA   9   76
 8.                     Alan STREET   8   72  <=== changed
 8.                  Matthew DAVIES   8   72  <=== changed
10.                   Alexei KOZLOV   8   65  <=== new
11.             Vakhtang MURVANIDZE   7   59

These relative standings remained unchanged after the next two skaters, Alexei Gruber and Michael Amentas, who finished farther down the list. Next up was Kyu-Hyun Lee, who received marks that gave him "wins" over Sandhu and all the skaters below him. However, the individual judges' placements of Lee compared to Davies and Street broke the tie on "judges in favor" between them: judge #5 voted for Street over Lee, while all the judges had Davies behind Lee. So now, the placements of Davies and Street flip-flopped:

                                    Wins JIF
 1.                 Derrick DELMORE  19  168
 2.                  Sergei DAVYDOV  18  158
 3.              Vitaly DANILCHENKO  17  159
 4.                  Taijin HIRAIKE  16  144
 5.                 Yosuke TAKEUCHI  15  129
 6.                    Kyu-Hyun LEE  14  124  <=== new
 7.                  Emanuel SANDHU  13  116
 8.                    Ben FERREIRA  11   93
 9.                     Alan STREET  10   89
10.                  Matthew DAVIES  10   88  <=== changed
11.                   Alexei KOZLOV  10   81
12.             Vakhtang MURVANIDZE   9   72

The next skater was Juraj Sviatko. His marks placed him even farther up the standings than Lee, but this time Davies picked up one judge on him, thanks to his generous marks from judge #6. So, Davies and Street were back in an absolute tie!

                                    Wins JIF
 1.                 Derrick DELMORE  20  177
 2.                  Sergei DAVYDOV  19  167
 3.              Vitaly DANILCHENKO  18  167
 4.                  Taijin HIRAIKE  17  151
 5.                   Juraj SVIATKO  16  137  <=== new
 6.                 Yosuke TAKEUCHI  15  133
 7.                    Kyu-Hyun LEE  14  126
 8.                  Emanuel SANDHU  13  119
 9.                    Ben FERREIRA  11   93
10.                     Alan STREET  10   89
10.                  Matthew DAVIES  10   89  <=== changed
12.                   Alexei KOZLOV  10   81
13.             Vakhtang MURVANIDZE   9   72

The next skater after this was Lukas Rakowski. Rakowski's marks were also high enough to place him well above the skaters involved in the tie, but once again Davies' high marks from judge #6 were enough to give him one more vote and break the tie with Street yet again, this time in his favor rather than Street's. Davies then retained this lead for the remainder of the competition.

Rakowski's placement is interesting in another way: judge #4 gave him the identical marks as Sviatko, and since the other 8 judges were split evenly, this also caused an absolute tie in determining the "win" in the head-to-head comparison between them. This in turn led to a tie in the number of "wins", which was broken by "judges in favor".

                                    Wins JIF
 1.                 Derrick DELMORE  21  186
 2.                  Sergei DAVYDOV  20  176
 3.              Vitaly DANILCHENKO  19  176
 4.                  Taijin HIRAIKE  18  159
 5.                   Juraj SVIATKO  17  142
 6.                  Lukas RAKOWSKI  17  139  <=== new
 7.                 Yosuke TAKEUCHI  15  136
 8.                    Kyu-Hyun LEE  14  128
 9.                  Emanuel SANDHU  13  123
10.                    Ben FERREIRA  11   94
11.                  Matthew DAVIES  10   90  <=== changed
12.                     Alan STREET  10   89  <=== changed
13.                   Alexei KOZLOV  10   81
14.             Vakhtang MURVANIDZE   9   72

World Juniors, Men's Free Skate

Another flip-flop occurred in the men's free skate, this time involving the two competitors from Canada. Emanuel Sandhu was initially ahead of Ben Ferreira, but Lukas Rakowski's marks gave all three of these skaters an identical number of "wins". Breaking the tie by "judges in favor" put Sandhu two places behind Ferreira with Rakowski between them.

In this instance, Sandhu was clearly hurt by a 16th-place ordinal from judge #5; no other judge had him lower than 12th. Again, this shows the fragility of the OBO-tie breaking mechanism with regards to inappropriate marks from a single judge.

World Juniors, Ladies Free Skate

In this competition segment, there was a flip-flop between Yea-Ji Shin and Huan Wang that also involved Anna Wenzel and Andrea Diewald in a four-way near-tie: Diewald "won" over Shin, who "won" over Wang, who "won" over Wenzel, who "won" over Diewald.

Europeans, Ladies Short Program

The ladies short program produced this very complex situation, involving a near-tie between three skaters. Because these skaters finished near the bottom of the rankings, this may not seem particularly critical to the outcome of the competition, but in fact it does have an affect on which skaters made the cut for the free skate, which would have been different under OBO than under the current ordinal system. This example also shows the extreme instability of the OBO tie-breaking mechanism in some circumstances.

After Katerina Blohonova skated, her marks put in next-to-last place over Jekaterina Golovatenko, like this:

                                    Wins JIF
 5.              Katerina BLOHONOVA   1   11  <=== new
 6.          Jekaterina GOLOVATENKO   0   11

These relative placements remained unchanged until Anina Fivian, the third of the competitors involved in the tie situation, skated. Her marks caused Golovatenko to move ahead of Blohonova, into a tie with Fivian.

                                    Wins JIF
 7.                    Anina FIVIAN   1   16  <=== new
 7.          Jekaterina GOLOVATENKO   1   16  <=== changed
 9.              Katerina BLOHONOVA   1   15  <=== changed

The next skater was Valeria Trifancova. Although her marks were generally much higher than the three skaters we're looking at, one of the judges had her behind Blohonova. This caused a three-way tie at the bottom of the standings:

                                    Wins JIF
 8.                    Anina FIVIAN   1   16
 8.              Katerina BLOHONOVA   1   16  <=== changed
 8.          Jekaterina GOLOVATENKO   1   16  <=== changed

After Noemi Bedo skated, the judges gave her marks that put her in last place. However, she won one judge each over Fivian and Blohonova, which caused Golovatenko to break out of the tie:

                                    Wins JIF
10.          Jekaterina GOLOVATENKO   2   25  <=== changed
11.                    Anina FIVIAN   2   24  <=== changed
11.              Katerina BLOHONOVA   2   24  <=== changed

Similarly, Salome Chigogidze won one more judge over Blohonova than Fivian, causing their tie to be broken, while Golovatenko remained in front:

                                    Wins JIF
15.          Jekaterina GOLOVATENKO   3   32
16.                    Anina FIVIAN   3   31
17.              Katerina BLOHONOVA   3   30  <=== changed

But, when Kaja Hanevold skated, the tie returned:

                                    Wins JIF
19.          Jekaterina GOLOVATENKO   4   40
20.                    Anina FIVIAN   4   36
20.              Katerina BLOHONOVA   4   36  <=== changed

And finally, it was broken again after Marta Senra skated:

                                    Wins JIF
23.          Jekaterina GOLOVATENKO   5   48
24.                    Anina FIVIAN   5   44
25.              Katerina BLOHONOVA   5   43  <=== changed

In summary, Blohonova was originally in front of Golovatenko and actually "won" the one-by-one comparison against her, but wound up finishing two places behind her and failed to finish in the top 24 as a result. Blohonova also moved in and out of a tie five times during the course of the event! In the real-life competition, it was Fivian who failed to make the cut rather than Blohonova.

Another very curious effect of the OBO scoring system is that it reversed the placements of the 2nd and 3rd place finishers in this competition segment from the actual results under the current ordinal system. The ordinals for the two skaters involved are:

Surya BONALY       5 4 2 2 5 2 1 1 4
Irina SLUTSKAYA    2 3 3 1 4 3 4 2 3

Under the current scoring system, Bonaly had a majority of five votes for second place, while Slutskaya had only three votes for second place or better. However, under the OBO simulation, Slutskaya was placed ahead of Bonaly, primarily because Slutskaya "won" over Bonaly by a 5-4 split in a direct comparison of their marks.

Europeans, Ladies Free Skate

In the ladies' free skate, Krisztina Czako's marks put her in a tie with Julia Lautova, with Yulia Vorobieva also having the same number of "wins" but one fewer "judges in favor":

                                    Wins JIF
 1.                 Irina SLUTSKAYA  21  182
 2.                Tanja SZEWCZENKO  20  181
 3.                 Elena LIASHENKO  19  167
 4.                 Julia SOLDATOVA  18  168
 5.                 Krisztina CZAKO  16  136  <=== new
 5.                   Julia LAUTOWA  16  136  <=== changed
 7.                 Yulia VOROBIEVA  16  135

However, when Surya Bonaly skated, Czako scored a "win" over her, while Lautowa did not, moving Czako definitely over Lautowa, who was moved into another tie with Vorobieva. Perhaps most surprisingly, in spite of Czako's "win" in the one-by-one comparison with Bonaly, she still wound up behind Bonaly in the final standings due to the OBO tie-breaking rules! In the actual competition using the current scoring rules, Bonaly finished behind Czako.

                                    Wins JIF
 1.                Maria BUTYRSKAYA  23  207
 2.                 Irina SLUTSKAYA  22  191
 3.                Tanja SZEWCZENKO  21  190
 4.                 Elena LIASHENKO  20  176
 5.                 Julia SOLDATOVA  19  175
 6.                    Surya BONALY  17  142  <=== new
 7.                 Krisztina CZAKO  17  141
 8.                   Julia LAUTOWA  16  139  <=== changed
 8.                 Yulia VOROBIEVA  16  139  <=== changed

Europeans, Men's Free Skate

In the men's free skate, Patrick Meier was initially ahead of Ivan Dinev, then tied with him, and then ahead of him again. Part of the problem here was that the Polish judge gave both competitors identical marks of 5.3/5.3 and the rest of the panel was equally split between them, so that they each scored a "win" over the other and were tied on "wins" throughout the entire competition; again, this allowed the unstable tie-breaking mechanism to come into play.

Olympics, Men's Short Program

The OBO simulation of this competition segment revealed three separate groups of flip-flops. The first involved Michael Weiss, Michael Shmerkin, and Szabolcs Vidrai. Shmerkin and Weiss both skated early in the draw, with Shmerkin winning the head-to-head comparison between them and therefore being ahead on "wins":

                                    Wins JIF
 2.               SHMERKIN, Michael   3   25
 3.                  WEISS, Michael   2   25  <=== new

Vidrai, skating later, then scored a "win" over Shmerkin while Weiss "won" over Vidrai, giving all three skaters an equivalent number of "wins". At this point, Vidrai and Weiss were also tied with 63 "judges in favor" with Shmerkin ranked behind both of them with 60.

                                    Wins JIF
 6.                VIDRAI, Szabolcs   7   63  <=== new
 6.                  WEISS, Michael   7   63  <=== changed
 8.               SHMERKIN, Michael   7   60  <=== changed

Eventually, however, Weiss pulled ahead of Vidrai on "judges in favor", picking up one additional judge over each of Dmitri Dmitrenko, Igor Pashkevitch, and Michael Tyllesen. Moreover, Weiss scored a "win" over Pashkevitch while Vidrai did not, giving him a clear lead on "wins" alone. In the end, Weiss, who had originally been placed behind Shmerkin, wound up finishing two places above him.

                                    Wins JIF
11.                  WEISS, Michael  17  153
12.                VIDRAI, Szabolcs  16  150
13.               SHMERKIN, Michael  16  144

This case is particularly interesting because it shows a flip-flop on "wins", not just one caused by skaters moving into a tie on "wins" and the order being changed by the "judges in favor" tie-breaker which is much more sensitive to fluctuations caused by strange marks from one or two judges.

The second flip-flop in this competition segment involved Igor Pashkevitch and Yamato Tamura. Pashkevitch had originally been placed behind Tamura on "wins", although he had a greater number of "judges in favor". When Pashkevitch scored a "win" over Viacheslav Zagorodniuk and Tamura did not, they were thrown into a tie on "wins" that was broken in Pashkevitch's favor. Both Pashkevitch and Tamura, incidentally, received a huge spread of ordinals -- from 8th to 18th for Pashkevitch, and 8th to 17th for Tamura!

Both of these two flip-flops involved the same group of six skaters who had scored "wins" over one another in a cyclical fashion: Weiss, Vidrai, Shmerkin, Pashkevitch, Tamura, and Zagorodniuk. Any possible ranking of these six skaters would have reversed at least one "win".

The final flip-flop involved Margus Hernits, Cornel Gheorghe, and Roman Skorniakov. Gheorghe initially led Hernits on "wins" and in the head-to-head comparison, but Hernits scored a "win" over Skorniakov while Gheorghe did not, throwing all three skaters into a tie with 6 wins apiece. At this point, the tie was broken by "judges in favor" in the order: Skorniakov, Hernits, Gheorghe. Skorniakov, however, later lost a judge to Kyu-Hyun Lee (who was well below the others in the overall standings), dropping him into a tie with Hernits. To complicate things further, all three skaters were thrown into an absolute tie when both Skorniakov and Hernits lost a judge to Anthony Liu (again, a skater who was placed well below the three tied skaters). In short, Gheorghe moved from being ahead of Hernits to being behind him to being tied with him; while Skorniakov wound up in a tie with two skaters he had initially led.

Olympics, Men's Free Skate

In the OBO simulation, Michael Tyllesen was initially ahead of Jeff Langdon on "wins", also having won the head-to-head comparison. However, Langdon later scored a "win" over Zhengxin Guo while Tyllesen did not, giving all three of these skaters 14 "wins" apiece. At this point, Guo was ahead with 124 "judges in favor" but Tyllesen and Langdon were tied with 120. Finally, Tyllesen picked up another judge over Steven Cousins, breaking the tie by moving him back ahead of Langdon.

A look at the ordinals shows that Langdon probably lost at least one place in the overall standings because of the single 15th-place ordinal he received from judge #2. In the OBO simulation, Langdon finished 11th in this competition segment, while in the actual competition under the current scoring rules, he was ranked 10th.

Olympics, Compulsory Dance 1

This situation involved a flip-flop between Irina Romanova & Igor Yaroshenko and Margarita Drobiazko & Povilas Vanagas. R&Y's marks initially put them ahead of D&V on "wins", although -- even at this point -- D&V had a greater number of "judges in favor". Skating two spots later, Sophie Moniotte & Pascal Lavanchy scored a "win" over D&V but not R&Y, giving all three teams 13 "wins" apiece. Breaking the tie by "judges in favor" flip-flopped the relative standings of R&Y and D&V, with M&L just ahead of both of them.

Here again it was obvious that a stray ordinal from a single judge had a disproportionate impact on the results -- in this case, it was the 5th-place ordinal given to Moniotte & Lavanchy from judge #2 which lifted them above the two other teams. (Most of their other marks were for 9th and 10th place.) In the actual competition under the current rules, M&L finished this competition segment in 10th place behind both of the other teams.

Olympics, Ladies Free Skate

In this competition segment, the OBO simulation showed a flip-flop between Alisa Drei and Anna Rechnio: the now-familiar situation where the marks given to a third skater (Laetitia Hubert) threw her into a tie on "wins" with two previous skaters (Drei and Rechnio), and breaking the tie by "judges in favor" reversed the relative placements of the two earlier skaters.

Summary

So, what can we conclude from these simulations?

First of all, it is very obviously the case that the OBO system does not eliminate flip-flopping within the standings. It is possible that it may even introduce more flip-flops than the current scoring system, although more statistical analysis of the two systems would have to be performed to determine this.

In situations where there is no clear consensus among the judges about the placements of skaters, the simulation shows it is quite common for the eventual results of the OBO system to place a skater behind another competitor in spite of "winning" a head-to-head confrontation with that competitor. Some of the scenarios discussed above even included instances where a skater "won" against the competitors who finished two and three places above him. This is troubling even in instances where it does not actually cause a flip-flop in the standings, because it indicates situations where additional flip-flops could have occurred if a different skating order had been drawn. It also means that OBO as a system is not internally consistent as to the significance of one-on-one "wins".

Another disturbing problem with the OBO system lies with using "judges in favor" as the sole tie-breaking mechanism. (Mathematically, "judges in favor" is actually equivalent to the "total ordinals" criteria that is used as the final tie-breaker in the current scoring system, after the "greater majority" and "total ordinals of majority" rules.) The problem is that a single judge who gives an ordinal that is greatly out-of-line with rest of the panel can wind up having a disproportionate effect should skaters wind up tied on the number of "wins". And ties on "wins" seem to be quite common in cases where the judging panel is split between a group of three or more skaters.

A mathematical note

The problem of combining rankings from multiple judges into a consistent ordering is the subject of a branch of mathematics called social choice theory. Mathematicians have shown that it is, in fact, impossible to construct a ranking system that preserves majority vote while avoiding flip-flops; it's a consequence of Arrow's Theorem.

From a theoretical point of view, OBO is therefore just as flawed in this way as the current system, or indeed as any other ordinal-based scoring system that could possibly be devised. The decision about which system is "better" has to be based on other criteria: perhaps computational simplicity and resistance to judging errors, which are both areas of comparative weakness for OBO, as we have seen.

Conclusions

It is clear that the OBO proposal does not represent an improvement over the current scoring system, considered purely on technical grounds. It meets neither of the supposed criteria for a new scoring system -- being easy to understand, and eliminating flip-flops. It is also inferior to the current system in that judging errors are more likely to affect the overall standings.

In addition, there are three disadvantages inherent in any change to the scoring system:

New computer software must be developed and/or purchased for use at all levels of competition, and accountants must be trained in its use. Accountants must also be trained in performing the hand calculations, and new educational and testing materials for certification of accountants must be developed. This would involve a significant cost in money and time to all ISU member federations.
Skaters, coaches, the media, and the public, who have had enough trouble grasping how the current scoring system works, must be educated about the new (and even more complex) rules.
The ISU must be prepared to deal with the controversies and bad publicity that will inevitably occur when competition results under the new rules differ from what would have happened under the old rules.

Given all of these costs and disadvantages, why is anybody in the ISU, and Ottavio Cinquanta in particular, pushing so strongly for the adoption of OBO? It appears that the scoring system has become a purely political issue that Cinquanta is using to curry favor with Juan Antonio Samaranch and the IOC, and that he and other ISU officials are using to consolidate or advance their own positions or power within that organization. It is hard to believe that any of the people who have been advocating OBO fully understand its implications, or are genuinely concerned about issues of fairness to the skaters, or care about making the scoring system more understandable to the public or increasing the public perception of the legitimacy of the sport.

It is also very disturbing that Cinquanta has attempted to suppress any methodical investigation of OBO by the technical committees and that the only official information about the proposed rules change that has been provided to the ISU members up until now has been in the form of vague generalities. It's my hope that the analysis and commentary contained in this article will make it possible for the ISU members to make a more informed decision on this issue at the upcoming ISU Congress.

Notes

This article incorporates revised and updated text from two of my earlier articles which were originally published on the Internet. These articles, along with the computer program, input data, and complete results for the simulations, are available on the web at http://www.frogsonice.com/skateweb/obo/.

I am indebted to Michael Stob (stob@calvin.edu) for supplying the connection to Arrow's Theorem.

Sandra Loosemore is a longtime skating fan and a regular contributor of commentary and reviews to the skating discussion groups on the Internet. She publishes The Figure Skating Page at http://www.frogsonice.com/skateweb/ and is the author of the Competitive Figure Skating Frequently Asked Questions List, a collection of reference and tutorial material about the sport that is regularly updated and published on the net. She is also a recreational figure skater.