The Truth About OBO

The Truth About OBO

By Sandra Loosemore (December, 1997)

The ISU is considering adoption of a new scoring system referred to as the "OBO" system, as a result of complaints that the current ordinal scoring system is too complicated and sometimes leads to "flip-flops" in the event standings -- situations where the relative rankings of two skaters change as a result of the marks given to a third skater.

In fact, the OBO system is not only even more complex than the current ordinal system, but it does not solve the flip-flop problem, either. The OBO system may even introduce flip-flops that would not have occurred under the current system.

This article begins with some background, followed by an overview of how the OBO system works and why it is more computationally complex than the current system. In the third part, some specific examples of flip-flops derived from a simulation of the results of the 1998 World Junior championships are presented. I conclude with some editorializing.

Background

The final results of the men's competition at the 1997 European championships were surprising to many people who did not understand how the ordinal-based scoring system used in figure skating works, or who were not paying close attention to the scoring as the event unfolded. Complicating the situation, the press covering the event (e.g., TV commentators such as Dick Button) seemed to be at a complete loss to explain the scoring system or results to the public. For more details about this, see my previous article at http://www.frogsonice.com/skateweb/obo/score-tech.shtml.

Following the European Championships, Ottavio Cinquanta, president of the International Skating Union, announced that he wanted a new scoring system adopted. Cinquanta has made repeated statements to the press that the new scoring system must not permit "flip-flops" in the standings, where the relative placements of two competitors who have already skated are changed by the marks given to a competitor who skates later, such as what happened at 1997 Europeans. (Most recently, at the 1997-98 Champions Series final in Munich, Cinquanta was quoted in press reports as stating that "If one skater is in front of another, he should remain there.") Also getting into the act was IOC chair Juan Antonio Samaranch, who made public statements to the effect that the system used to score figure skating at the Olympics must be understandable to the public.

In fact, the ISU's own regulations did not allow changes to the scoring system to be adopted in time for the 1998 Olympic Winter Games. In the meantime, while various other alternatives were presented, Mr. Cinquanta focused on the "OBO" proposal for a new scoring system presented to the ISU by Andreas Sigurdsson and sponsored by the DEU (the German federation). The ISU technical committees were instructed to allow the DEU to test the OBO system at the Nebelhorn Trophy competition in August, 1997. The results of that test were inconclusive and subject to dispute as to what a "change of position" really was. Rumors then began swirling at the 1998 World Junior championships that the marks awarded there would be recalculated under OBO and presented to the technical committees for further review during meetings to be held at the Champions Series final.

OBO: What it is, how it works

OBO stands for "one-by-one". The general idea behind this scoring system is to compare the skaters pairwise and rank them according to which has the most total "wins" in these pairwise comparisons. Some people may find the idea of ranking skaters according to a "who-beat-who" criteria appealing, but this is definitely not how the OBO system works. It is quite possible under OBO for skater X to place ahead of skater Y even if Y beat X in a head-to-head comparison of marks, for example.

The OBO system does not introduce any changes to the marking used by the judges. Under OBO it is also convenient to compute ordinals in the same way as for the current scoring system since this simplifies the later computations somewhat. However, the manner of using the ordinals from the different judges to calculate standings is completely different than for the current system. Although statements in the press suggest this is just a minor change to the current system to prevent flip-flops, the actual differences in standings between the two systems can be dramatic.

If there are N skaters participating in the event, the OBO system requires building an N-by-N table, with a row and column for each skater. The entries in the table are filled in by comparing each skater against all the others. In comparing skater X against skater Y, there are two pieces of information that have to be filled in each table entry: whether X skater has "won" a majority of judges when compared to Y, and the number of judges that voted for X over Y (this is called "Judges in Favor", or "JiF"). Then the total number of "wins" for each skater is added up, as is the total number of "Judges in Favor". The skaters are ranked by total wins with total JiF used as the tie-breaker.

Here's an example involving a small event with only 6 competitors (derived from the actual marks given to the top 6 finishers in the free skating in the men's event at 1997 Europeans). The first step is computing the ordinals in the usual way. Suppose this works out to give us:

A   1  1  1  1  1  2  1  1  1
B   3  2  5  2  3  3  5  6  6
C   5  5  4  4  2  4  2  2  3
D   4  3  3  6  4  6  4  3  2
E   2  4  2  3  6  5  3  4  5
F   6  6  6  5  5  1  6  5  4

The OBO comparison table might look like this:

    ||  A  |  B  |  C  |  D  |  E  |  F  ||  total wins  ||  total JiF
----++-----+-----+-----+-----+-----+-----++--------------++-------------
 A  ||     | 1/9 | 1/9 | 1/9 | 1/9 | 1/8 ||     5        ||    44
----++-----+-----+-----+-----+-----+-----++--------------++-------------
 B  || 0/0 |     | 0/4 | 1/5 | 0/4 | 1/6 ||     2        ||    19
----++-----+-----+-----+-----+-----+-----++--------------++-------------
 C  || 0/0 | 1/5 |     | 1/5 | 1/5 | 1/8 ||     4        ||    23
----++-----+-----+-----+-----+-----+-----++--------------++-------------
 D  || 0/0 | 0/4 | 0/4 |     | 0/4 | 1/7 ||     1        ||    19
----++-----+-----+-----+-----+-----+-----++--------------++-------------
 E  || 0/0 | 1/5 | 0/4 | 1/5 |     | 1/6 ||     3        ||    20
----++-----+-----+-----+-----+-----+-----++--------------++-------------
 F  || 0/1 | 0/3 | 0/1 | 0/2 | 0/3 |     ||     0        ||    10
----++-----+-----+-----+-----+-----+-----++--------------++-------------

So in this case the results would be A, C, E, B, D, F. (Under the current scoring system, the results were A, B, C, D, E, F; so you can see that the OBO system does indeed produce results that are quite different.)

Aside from any questions regarding the effectiveness of the OBO scoring system, such as whether or not it is resistant to judging errors or prevents flip-flops in the standings during the course of an event, it is immediately obvious that it is much more complex from a computational point of view than the current system. The current system requires only a linear scan of the ordinals given to each competitor to compute the ranking criteria (the majority, total ordinals of majority, and so on). The OBO system, on the other hand, requires an order N-squared comparison followed by a linear scan.

The additional computational complexity may not appear to be a big issue in the age of ubiquitous computers, but in fact many skating fans who attend competitions can and do work out the results by hand using the scoring sheets provided in the program booklet. At competitions that do not have electronic displays of running standings, fans often work out the results by hand long before the official results are posted. Under the OBO system, doing this would at best be very impractical. Certainly, at an event like the World Championships when there are up to 30 competitors, a very large sheet of paper would be necessary to work out or display the 30-by-30 table of results, and it would be extremely tedious and error-prone to have to do at least an additional 3,915 comparisons by hand, along with repeated additions of 30 columns of numbers, to fill in the table.

A further problem regarding the complexity of the OBO system is one of the issues that initially led to the search for a new scoring system: the complaint that the scoring system was too complex for the TV-viewing public to understand or for the media to explain. Introducing a scoring system that is even more complex, and that in many instances produces results that are just as counter-intuitive, is certainly not going to solve this problem.

So what about flip-flops?

In fact, the OBO system does not prevent flip-flops in the event standings -- instances where the relative placements of two skaters change because of the placements given to a third skater by the judges.

To show how the OBO system might work in a real-life competition situation, a simulation was performed using the marks and skate orders from the 1998 World Junior competition, available on the web at http://www.wige.de/Skating/cgi-bin/subev.idc?eevt_id=9. (Recall that the OBO system doesn't change the interpretation of the judges' marks from the current rules, only the manner in which the results are computed from the marks.) This simulation shows that, under the OBO system, flip-flops would have occurred in at least four phases of the competition.

Ladies Qualifying Round B

In the OBO simulation, Julia Sebestyen was ahead of Angela Nikodinov with a greater number of "Wins" until Anina Fivian skated. At this point, all three of the skaters were thrown into a tie with an equal number of "Wins", and breaking the tie by "Judges in Favor" ranked the skaters in order Fivian, Nikodinov, and Sebestyen, so that Sebestyen and Nikodinov "flip-flopped" from their previous standings.

It is also interesting to observe that Nikodinov actually scored a head-to-head "win" over Fivian, and Sebestyen "won" over Nikodinov -- exactly the opposite of their final placements in the OBO simulation.

Pairs Short Program

In the OBO simulation, after 9 teams had skated, Natalie Vlandis & Jered Guzman and Sabrina Lefrancois & Nicolas Osseland were in second and third places, respectively, in a tie by total "Wins" but with V&G slightly ahead by "Judges in Favor". However, after Jacinthe Lavriere & Lenny Faustino skated, the placements of V&G and L&O flip-flopped because L&O received more "Judges in Favor" against L&F than did V&G. This happened even though L&F placed well below (5th) both of the pairs who were affected by the flip-flop, and is an example of how erratic marks given by a minority of judges can have a major impact on the overall placements of the competitors under the OBO system.

In this event as well, there were instances where the overall placements of the competitors concerned did not reflect the "wins" in head-to-head comparisons: e.g., Vlandis & Guzman actually "won" over Lefrancois & Osseland.

Original Dance

Jamie Silverstein & Justin Pekarek flip-flopped with Natalia Romaniuta & Danil Barantsev after Aleksandra Kauc & Filip Bernadowski skated. This was a case where the skaters were initially tied on "Wins" but where Silverstein & Pekarek had a greater number of "Judges in Favor". When K&B skated, they "won" against S&P but not R&B, which broke the tie in "Wins" between the latter two teams. However, even though R&B wound up with more "Wins", as the placements for the remaining skaters in the competition were computed, R&B continued to have fewer "Judges in Favor" than S&P. Again in this competition segment, it was observed that a single judge who gave marks that were out of line could have a significant effect in the overall standings due to the "Judges in Favor" tie-breaker. For example, S&P's higher "Judges in Favor" count could be attributed to the scattering of 5th, 6th, and 7th place votes they received although they had a majority only for 10th place. Meanwhile, R&B's one 11th-place vote among their majority for 8th place essentially meant that the one judge cost them three "Judges in Favor" points.

Men's Short Program

This was probably the most interesting and complex of the flip-flops revealed by the OBO simulation, because it involved numerous skaters, two flip-flops, absolute ties, and placements being changed as a result of marks given to skaters who placed nowhere near the skaters affected. It is probably easiest to present this scenario by showing the standings of the skaters as the event unfolded in the simulation.

The fun started when Alexei Kozlov skated. Before this, the standings for the top 10 were:

                Wins
 1  Delmore      15
 2  Davydov      14
 3  Danilchenko  13
 4  Hiraike      12
 5  Takeuchi     11
 6  Sandhu       10
 7  Ferriera      9
 8  Davies        8
 9  Street        7
10  Murvanidze    6

Kozlov was given a mixed bag of marks that gave him "wins" over Ferriera and Davies but not Street or Murvanidze. This meant that Davies, Street, and Kozlov now all had 8 "wins", and moreover, Davies and Street were in an absolute tie with the same total "Judges in Favor". Note that at this point Kozlov is behind two of the skaters he beat in head-to-head competition, and ahead of one who beat him!

                Wins  JiF
 1  Delmore      16
 2  Davydov      15
 3  Danilchenko  14
 4  Hiraike      13
 5  Takeuchi     12
 6  Sandhu       11
 7  Ferriera      9
 8  Davies        8   72
 8  Street        8   72
10  Kozlov        8   65
11  Murvanidze    7

These relative standings remained unchanged after the next two skaters, Gruber and Amentas, who finished farther down the list. Next up was Lee, who received marks that gave him "wins" over Sandhu and all the skaters below him. However, the individual judges' placements of Lee compared to Davies and Street broke the tie on "Judges in Favor" between them: judge #5 voted for Street over Lee, while all the judges had Davies behind Lee. So now, the placements of Davies and Street flip-flopped:

                Wins  JiF
 1  Delmore      19
 2  Davydov      18
 3  Danilchenko  17
 4  Hiraike      16
 5  Takeuchi     15
 6  Lee          14
 7  Sandhu       13
 8  Ferriera     11
 9  Street       10   89
10  Davies       10   88
11  Kozlov       10   81
12  Murvanidze    9

The next skater was Sviatko. His marks placed him even farther up the standings than Lee, but this time Davies picked up one judge on him, thanks to his generous marks from judge #6. So, Davies and Street were back in an absolute tie!

                Wins  JiF
 1  Delmore      20
 2  Davydov      19
 3  Danilchenko  18
 4  Hiraike      17
 5  Sviatko      16
 6  Takeuchi     15
 7  Lee          14
 8  Sandhu       13
 9  Ferriera     11
10  Street       10   89
10  Davies       10   89
12  Kozlov       10   81
13  Murvanidze    9

The next skater after this was Rakowski. Rakowski's marks were also high enough to place him well above the skaters involved in the tie, but once again Davies' high marks from judge #6 were enough to give him one more vote and break the tie with Street yet again, this time in his favor rather than Street's. Davies then retained this lead for the remainder of the competition.

Rakowski's placement is interesting in another way: judge #4 gave him the identical marks as Sviatko, and since the other 8 judges were split evenly, this also caused an absolute tie in determining the "win" in the head-to-head comparison between them. This in turn led to a tie in the number of "Wins", which was broken by "Judges in Favor".

                Wins  JiF
 1  Delmore      21
 2  Davydov      20
 3  Danilchenko  19
 4  Hiraike      18
 5  Sviatko      17   142
 6  Rakowski     17   139
 7  Takeuchi     15
 8  Lee          14
 9  Sandhu       13
10  Ferriera     11
11  Davies       10   90
12  Street       10   89
13  Kozlov       10   81
14  Murvanidze    9

Summary

So, what can we conclude from these simulations?

First of all, it is very obviously the case that the OBO system does not eliminate flip-flopping within the standings. It is possible that it may even introduce more flip-flops than the current scoring system, although more statistical analysis of the two systems would have to be performed to determine this.

In situations where there is no clear consensus among the judges about the placements of skaters, the simulation shows it is quite common for the eventual results of the OBO system to place a skater behind another competitor in spite of "winning" a head-to-head confrontation with that competitor. The men's short program scenario above even included an instance where a skater "won" against the competitors who were two and three places above him. This is troubling even in instances where it does not actually cause a flip-flop in the standings, because it indicates situations where additional flip-flops could have occurred if a different skating order had been drawn. It also means that OBO as a system is not internally consistent as to the significance of one-on-one "wins".

Another disturbing problem with the OBO system lies with using "Judges in Favor" as the sole tie-breaking mechanism. The problem is that a single judge who gives an ordinal that is greatly out-of-line with rest of the panel can wind up having a disproportionate effect should skaters wind up tied on the number of "wins". And ties on "wins" seem to be quite common in cases where the judging panel is split between a group of three or more skaters.

Conclusions

It is not clear to me why anyone connected with the ISU is spending any more time on evaluating or advocating the OBO system, since it meets neither of the supposed criteria for a new scoring system -- being easy to understand, and eliminating flip-flops. It is also inferior to the current system in that judging errors are more likely to affect the overall standings. It certainly seems better to leave well enough alone and concentrate on educating the public and the media about the current scoring system or seeking out improved ways to present the results so that they are more understandable to the public.

It is also not clear to me why Ottavio Cinquanta released a statement to the press during the Champions Series Final in Munich, announcing the adoption of the OBO system as the solution to the "flip-flop" problem before the technical committee meeting during the event had even finishished preparing its report. If in fact the ISU's technical committees did use 1998 World Juniors as a test case, they must have become aware of the same problems and had much the same conclusions about them. From the press reports, it appears that Cinquanta also asserted that the OBO proposal had the backing of the ISU council, when in fact the proposal apparently has only a minority of clear supporters among the ISU council and technical committee members. By making premature and misleading claims on behalf of the OBO system, it seems like Cinquanta has merely created more confusion about how skating is scored.

Sandra Loosemore is a longtime skating fan and a regular contributor of commentary and reviews to the skating discussion groups on the Internet. She publishes The Figure Skating Page at http://www.frogsonice.com/skateweb/ and is the author of the Competitive Figure Skating Frequently Asked Questions List, a collection of reference and tutorial material about the sport that is regularly updated and published on the net. She is also a recreational figure skater.