Code of Points Considered Harmful

by Sandra Loosemore
May, 2003

Introduction

In response to the scandal at the Salt Lake City Olympic Winter Games in 2002 where a figure skating judge admitted to being involved in a plot to "fix" the result of the pairs competition, ISU President Ottavio Cinquanta has aggressively pushed for the adoption of a radical revision of the judging system used in figure skating. Cinquanta's new system utilizes cumulative marking based on a "code of points" for every element instead of comparative judging on a 6.0 scale.

Although Cinquanta repeatedly described the new system as a "project, not a rule" when it was proposed at the ISU Congress in the summer of 2002, an outline of the system -- including some clauses not specifically approved by the Congress -- later appeared in the 2002 edition of the ISU General Regulations as Rule 121. At its meeting following the 2003 World Championships, the ISU Council voted to adopt the new "project-not-rule" system in the fall of 2003 at the Nebelhorn Trophy, the six Grand Prix competitions, and the ISU Grand Prix Final. At this time, the ISU released ISU Communication no. 1207, the first public document containing any specific details of the proposed "Code of Points".

In this article, we examine some features of the code of points system and compare it to the current rules. It may be helpful for readers to first try to read the actual text of ISU Communication no. 1207, but for those who have limited patience for technical language, here are some of the key points of the proposed new judging system:

The element tables are incomplete

ISU Rule 121, as it appears in the 2002 General Regulations, states that "the ISU will issue a Scale of Value for each and every element that compose the events of Figure Skating." However, it is clear that ISU Communication no. 1207 does not include "each and every element".

The most obvious omission is that the tables of jumps for singles and pairs are incomplete. For example, while the tables include the six most common single jumps -- toeloop, salchow, loop, flip, lutz, and axel -- they do not include jumps such as the walley, inside axel, one-foot axel, split flip or lutz, half loop, 1.5 flip, and so on, which are quite commonly used as linking elements or highlight moves in a program. Also missing are less common jumps which have been performed on occasion in the past: a toeless lutz, double walley or double one-foot salchow, for example.

Are skaters who attempt these elements to get no credit at all for doing them? If a skater does a one-foot axel/triple salchow combination, does this even count as a jump combination since the one-foot axel is not officially listed as a jump? Alternatively, if these elements are indeed treated as jumps and count against the maximum numbers of permitted jumps in the program, why would any skater add an inside axel or split lutz to their program if it means they must take out a triple axel or a triple lutz?

It is also very peculiar that a layback spin does not exist in the published code of points, given that it is a required element in the short program for ladies. The tables list only upright, camel, sit, and combination spins. The flying spins apparently comprise only the flying upright, flying camel, and flying sit spin, but not the flying reverse sit spin, "death drop" (or open axel sit spin), and butterfly spin. There are no entries for combination spins that commence with a jump or that include a jump transition.

Another curious omission is that a throw lutz jump is apparently considered not to exist in pair skating. Yet, there have been real instances of pairs performing this move in the past.

It seems questionable that "moves in the field" -- spread eagles, spirals, hydroblading, and other edge moves -- are not considered elements at all unless they are incorporated into one of the required step sequences. Instead, such elements are lumped with "Transitions", even though many skaters use them as highlight moves in their own right.

If the omissions are simply oversights or mistakes in the specification of the system, it is certainly unfair to skaters, coaches, and choreographers that they have this unfinished system being foisted upon them for some of the most important events of the year. Even if the omissions are eventually corrected by the ISU, skaters have to plan and choreograph their programs for the upcoming season based on the information they have available to them now.

Another concern is that giving credit only to elements specifically listed in the code of points gives skaters little or no motivation for inventing new elements or including very unusual elements in their programs. While the caller has the discretion to give competitors credit for a 2-point "bonus" element, only one such bonus can be awarded per program and the value of this bonus may not be large enough to convince skaters to attempt unlisted elements that are particularly difficult. Moreover, it is not clear what kind of element could qualify for the "bonus", since the code of points system establishes new maximum limits on the number of jumps, spins, and step sequences permitted in a program.

There is not enough information about how non-jump elements are classified

Spins and step sequences are listed in the code of points as "Level 1", "Level 2", and "Level 3". However, there is absolutely no information given about the criteria the caller uses to assign these levels. Is it completely arbitrary and up to the judgement of the individual caller?

Even if this is a mistake or oversight that is later corrected by the ISU, this is not fair to skaters because they do not have enough information to know in advance how the technical content of their programs will be evaluated, or to construct their programs in such a way as to obtain maximum credit under the new judging system.

The point values for some elements don't make sense

The base values assigned to some of the elements in the code of points appear to be extremely arbitrary and don't always correspond to common perceptions about the relative difficulty of various elements.

Perhaps the most egregious example of this is in the way jump combinations are scored. The base value of a jump combination is assigned simply by adding together the base values of its constituent jumps, while everyone who skates knows that it is more difficult to do jumps in combination than to do them individually. This method of assigning a base value to a jump combination also does not take into consideration the fact that the order of the jumps in the combination affects the difficulty of the combination as a whole -- for example, a double toe/triple toe combination is much more difficult than a triple toe/double toe.

Even more ridiculous is the way that jump sequences are marked in the new system. The base value of a sequence is determined by adding together the base values only of its two most difficult jumps, which completely takes away any motivation for skaters to include more than two jumps in a sequence. On top of that, the base value of the sequence is then multiplied by 0.8 -- so that skaters are effectively penalized for doing jumps in sequence instead of as individual jumps. This completely flies in the face of common sense.

Finally, the "grade of execution" adjustment applied to the base value of the combination or sequence is taken from the table for the most difficult jump element only, instead of the sum of the values for the jumps in the combination. This again has the paradoxical effect that a skater will receive less credit for a well-done combination than for doing the constituent jumps equally well in isolation.

The absurdities in the code of points aren't restricted to jumps. For both singles and pairs, straight-line, circular, and serpentine step sequences are all assigned the same base values, even though it's widely considered that circular and serpentine patterns are more difficult than straight-line because of the significantly greater ice coverage and the need for more steps in the sequence to cover the ice. Similarly, upright, camel, and sit spins are all assigned the same base values even though anyone who skates knows that a camel spin is significantly more difficult than an upright spin.

Among the elements for pairs, the two inside death spirals are given the same base values, as are the two outside death spirals. But most people who are familiar with pair skating would probably classify the forward inside death spiral as being easier than the backward inside, and the forward outside death spiral as being much more difficult than any of the others.

Even where the elements seem to be assigned base values that rank them correctly in order of difficulty, one has to question the correctness of the absolute values and the process used to assign the base values. For example, who decided that a double axel is 1.74 times as difficult as a double lutz, but a triple axel is only 1.23 times as difficult as a triple lutz? One must not make the mistake of assigning precision to the values in the code of points where there is no underlying accuracy in what the values represent. In other words, the decisions regarding the values assigned to elements are just as arbitrary and subjective as in the current judging system.

There is too much value given to failed jumps

Failed jump elements seem to be given far too much credit in the code of points. For example, a skater who completely messes up a quadruple salchow will still get 5.5 points just for making the attempt, which is more than they will get for executing a clean triple salchow, or performing the most difficult and well-executed spin combination imaginable. Surely this will do nothing other than encourage skaters to include jumps in their programs that they have little hope of executing cleanly or well. Under the current judging rules, it is fairly clearly understood that skaters deserve little or no credit for jumps marred by a fall or other serious errors, such as landing an underrotated jump on two feet, or putting a hand down or stepping out to avoid falling.

Rewarding skaters for failed jumps does not seem to be in the best interests of the sport. Television audiences will be subjected to watching all of the falls and failures, and this is not likely to be good for viewership. Perhaps some people enjoy watching short-track races for the sake of the crashes, but the same cannot usually be said of fans who watch figure skating.

The requirements for a well-balanced program are being changed

The ISU regulations include a list of requirements for a well-balanced free skating program in each discipline. Under the current rules for singles, the number of jumps is not limited, other than the restriction on repeating triple and quadruple jumps. Skaters must do at least one jump combination or jump sequence but not more than three, with no restriction on the number of jumps permitted in a combination. A minimum of four spins and two step sequences are required.

Communication no. 1207, on the other hand, changes these requirements to limit the number of "jump elements" (individual jumps, combinations, or sequences) to 8 for men and 7 for ladies, which must include an axel-type jump. There is a maximum of only two combinations or sequences and the additional restriction that a jump combination cannot consist of more than three jumps. A maximum of four spins and two step sequences are permitted.

What will the effect of these changes be? It's fairly obvious that imposing tighter restrictions on program content will result in skaters performing more cookie-cutter programs, all with similar elements and construction designed to gain maximum point value. The long program will become more like an extended short program rather than a true free skating program.

Skaters will have little motivation to include single and double jumps, such as axel variants, as highlight elements in their programs, since these jumps will count against their maximum number of permitted jump elements and will get them few points even compared to unsuccessful triple and quadruple jump attempts. The restrictions on jump combinations will limit skaters' creativity with multi-jump combinations and sequences, and indeed, with the devaluation of jumps in sequence, it is unlikely that any skater would deliberately plan to include a jump sequence in their program. Likewise, skaters will have no motivation to include additional spins or step sequences in their programs, since they will receive no credit for them. Instead, we may very well see skaters spending more of their programs standing around posing in front of the judges instead of executing skating elements which would add difficulty and variety to the overall performance but for which they would receive no credit under the new rules.

The restrictions on jumps will also have some more subtle effects on program construction. For example, under the existing ISU rules, many elite ladies competitors plan a program with 7 triple jumps plus a double axel. Under the new judging system, since a maximum of 7 jump elements are allowed, this effectively will force skaters who want to continue to do 7 triples to attempt a triple/triple combination.

Finally, a very big problem with changing the requirements for a well-balanced program is that although the new code of points system may be used at the Grand Prix events in the upcoming season, at other competitions -- including skaters' national championships and the World Championships -- they will still be judged by the existing ISU regulations and technical standards. A program that is designed to obtain maximum credit under the code of points system will not also gain maximum credit under the existing rules, and vice versa. Skaters, coaches, and choreographers are having a difficult time understanding the changes to skating standards forced by adoption of the code of points system in isolation; it's even more intolerable and unfair that skaters should have to cope with training programs to meet two incompatible sets of technical requirements.

The balance between technical merit and presentation is being changed

Let us now consider the relationship between the marks given to individual elements and the five subjective "Program Components" criteria, and how this compares to the equal weighting of the technical merit and presentation marks under the current ISU rules.

As a concrete example to work with, consider a program that might be performed by an elite senior lady which has been constructed to fit the new requirements for a well-balanced program. Such a program might contain a triple lutz/triple toe combination, a triple loop/double loop combination, solo triple lutz, flip, salchow, and toe jumps, and a double axel; two spin combinations, a flying spin, and a spin with only one position; a spiral step sequence and one other step sequence. We'll assume that the second lutz, toe loop, and axel are done in the second half of the program so that the scores for those elements are multipled by the bonus factor of 1.1. Let us further assume that the spins and step sequences are all level 3, since we are trying to consider the equivalent of a 6.0-worthy performance. Adding up the maximum possible values for all these elements, we see that the jumps are worth 64.99 points, the spins 18.4 points, and the step sequences 12.8 points, for a total of 96.19 possible points.

The five "Program Components" are each marked on a scale of 0-10. As an additional wrinkle, for the ladies free skate, the Program Components marks are multiplied by a factor of 1.6, which means that they contribute 80 points altogether to the overall score. Therefore we see that the balance of the marking is tipped slightly in favor of the individual technical elements rather than the overall Program Components.

Summary of point allocation for sample program
Individual elementsJumps64.99
Spins18.4
Step sequences12.8
Total individual elements96.19
Program componentsSkating Skills16.0
Transitions16.0
Performance/execution16.0
Choreography16.0
Interpretation16.0
Total program components80.0
Total176.19

At a first glance, this may not seem like a radical departure from the equal balance between technical merit and presentation in the current ISU rules. But looking more closely at the criteria for the individual "Program Components", we can see that the "Skating Skills" and "Transitions" components contain criteria that are currently considered under the technical merit mark rather than the presentation mark. This means that under the code of points system, for this example program the technical merit criteria would total 128.19 points as compared to only 48 points for the presentation criteria, a far cry from the current equal weighting.

Even more ridiculously, the jumps alone carry significantly more weight than the three presentation components combined -- 64.99 points for the jumps versus 48 points for presentation. Furthermore, skaters can further drive up the technical merit portion of the score by attempting more difficult jumps while the contribution of the presentation portion remains fixed. This will only result in the imbalance becoming more pronounced.

It seems somewhat incredible that a change of this magnitude in the fundamental balance of the marking is being implemented with so little opportunity for discussion and feedback from the skating community. What evidence is there of any consensus among skaters, coaches, fans, television broadcasters, and other stakeholders that this is truly the correct direction to move the sport? For example, does anyone really think that devaluing the artistic component of skating while rewarding skaters for failed triple and quadruple jump attempts is going to attract audiences to the sport and drive up TV ratings? What about the increased risk of injury to skaters from a greater emphasis on jumping? In fact, the true situation seems to be that the key stakeholders have not even been provided with sufficient information to recognize and understand the full consequences of the adoption of the code of points system before the ISU Council put it in place.

The "callers" have too much power and no accountability

Moving on, let us now consider some other aspects of the implementation of the code of points system.

As noted in the summary description of the new system, identification of the various technical elements is the responsibility of a "caller", also referred to as a "technical specialist". These persons are intended to be recruited from coaches, former elite skaters, and other skating "experts".

The "caller" has a great deal of responsibility in the new judging system. Aside from correct identification of jumps, the "caller" has the responsibility for assigning levels to spins and step sequences, which can have as much or more impact on the amount of credit skaters get for them as the grade of execution assigned by the judges. This is particularly critical in ice dance, where all of the elements listed for free dancing in the code of points are categorized by level at the discretion of the caller.

The potential for error is obvious. Under the current judging system, there are nine pairs of eyes watching each performance, and nine judges making independent decisions about the identification and difficulty of the elements. Even if one judge misses an element or gets it wrong, the other eight are still likely to get it right and the overall result will not be affected. But under the code of points system, this is all reduced to one pair of eyes and one person's judgement. And we already know, from seeing the work of similarly qualified persons employed as television commentators, how frequently even "experts" mis-identify jumps or even completely miss elements! Furthermore, callers can (and most certainly will) be biased, and judges cannot override the callers.

There are no published procedures for establishing the qualifications of the "callers", evaluating their performance, or making them accountable for their work. Judges, on the other hand, have to be appointed by their national federations and pass a judge's examination, and their work is reviewed by the event referee and the appropriate Technical Committees. Given that "callers" have as much power as judges or more under the new code of points system, they must be made subject to similar procedures and not simply treated as paid consultants.

The new system still utilizes "secret judging" with no accountability

The ISU has taken a great deal of criticism from the media, skaters, fans, and coaches alike over the interim "secret judging" system that was adopted in 2002. The use of a secret and random computerized draw to select only a subset of the judges on the panel whose marks will "count" and particularly the lack of any identification of which judge gave which marks -- and the subsequent lack of public accountability of the judges for their actions -- have been characterized in the press as a public relations disaster for the ISU.

ISU President Cinquanta has attempted to downplay these concerns by drawing attention away from the interim system. Instead, he has been emphasizing all the supposedly wonderful features of the code of points system and urging its adoption as a long-term replacement for the interim system as soon as possible.

What Cinquanta has not been saying is that the code of points system, too, utilizes "secret judging". There will still be a random computerized draw to select a subset of the judges on the panel. None of the marks will be identified by the judge who assigned them. There will still be no discussion of the judging in the event review meeting. There will still be no public accountability in the judging.

The computation method is statistically unsound

Aside from the issues of accountability, the "secret judging" aspect of the code of points system shares another misfeature with the ISU's interim judging system: the mistaken belief that choosing only a random subset of the judges' marks to count makes the result more accurate than using the marks of the entire panel. From a statistical point of view, this is completely incorrect. As long as the ISU is going to pay the expenses to put 14 judges on the panel, they will get the most accurate results by using the marks of all 14 judges. This is true even if some of the judges are biased or incompetent.

In the code of points system, this flaw is magnified by using a "trimmed mean" method to determine the overall score of the panel: dropping the two high and two low marks, and then averaging the rest. It is well known in statistics and in past statistical studies of scoring systems for figure skating and other judged sports that the trimmed mean is less accurate and more prone to bias and manipulation than computation methods that use the median mark or a majority system.

Even more fundamentally, when the overall goal of the scoring method is to assign a ranking -- first, second, third, and so on -- studies have shown that humans can achieve more accurate results by using a scoring system that is itself based on rankings (or ordinals) than on absolute point values. By abandoning the current ordinal system for a new scoring system based on point values, the ISU is therefore choosing an inferior and less accurate scoring method.

The system has not been adequately tested

The code of points system that the ISU has approved for use at major competitions this season has never been tested in its current form under similar competition conditions.

Last fall, the ISU conducted two so-called "tests" of the code of points system, one at Skate Canada and one at the Junior Grand Prix Final. However, the system was changed as a result of these tests, so that the code of points being implemented now is not the same one that was used for these tests. In addition, the tests performed last year were on performances constructed to meet the ISU's existing standards for judging and well-balanced programs, not the new standards specified in Communication no. 1207. Another difference between the tests and the actual application of the system is that both of the events where the system was tested last year were competitions with small fields of elite competitors; by contrast, Nebelhorn Trophy, where the first official use of the code of points system will occur, is often a very large competition with 25 or more competitors entered in each of the singles events.

Aside from the questions about the validity of the test conditions, the results from last year's tests have never been made public for discussion, review, or feedback by the skating community as a whole. One might consider that the system has been "tested" in the sense that it has been demonstrated that the judges can use the computer interface and the computations can be performed to produce a result for a competition. But what is the purpose of a test if not to determine whether the system produces the correct result under controlled conditions? Skaters, coaches, and other critical stakeholders in the sport have had no opportunity to compare the two judging systems for themselves to make a realistic, practical assessment of how the adoption of the code of points system will affect them directly, or the direction of the sport as a whole.

The new system is overly reliant on computer technology

One of the "selling points" the ISU has been using to promote the code of points system is its use of sexy high technology -- touch-screen displays, instant replay, everything computerized. But in fact this is a serious problem with the new system, not a feature!

A previous article has already covered in detail many of the problems that have plagued the scoring computers at recent figure skating competitions: everything from system crashes, to producing results that are obviously incorrect, to producing two sets of different results, to uncorrected data entry errors. Given that the software controlling the ISU's prototype code of points computer system has been built by the same contractor that produced the previous scoring software, there is little reason to hope it will have any fewer bugs and glitches.

Because of the "secret judging", the raw marks input by the judges will not be available for public inspection, and there will be no way for the public to verify that the computations are being carried out correctly by the scoring software. The secrecy and complexity of the new system will combine to make it impossible for anyone to verify that the judges are not cheating and that the scoring software is not full of bugs.

Moreover, the consequences of a computer malfunction become much more serious under the new system because its implementation is so completely dependent on computer technology. What happens if there is a computer crash during a skater's performance that disables the judges' touch-screen displays? Will the skater have to be stopped by the referee until the computer can be restarted? Otherwise, without the computer-driven displays, how are the judges supposed to know the elements identified by the caller, or enter their marks? With marks for the elements being entered directly into the computer during the performance instead of onto a paper scoresheet, what will happen if a computer crash wipes out some of the data?

A further problem is the sheer cost of all this computer equipment, proprietary software, and the technical specialists required to set it up and operate it. The ISU seems to be paying no attention to whether the code of points system can practically be adapted to hand computation or use without touch-screen displays and the like. Cost may not be a big problem for the ISU when they consider only a handful of high-profile competitions each year, but those are only the proverbial tip of the iceberg in the skating world.

In the United States alone, there are hundreds of club competitions held each year. Thousands of skaters participate in the qualifying competitions which lead to the national championships and, ultimately, the World Championships and other events controlled by the ISU. It would be wildly expensive and impractical to provide all of the high-tech equipment needed to support the code of points system at all of these events. It's quite probable that some smaller national skating federations could not support the cost of the use of the computerized system even at their national championships only.

The alternative is that skaters will have to continue to compete under the old rules or something like them at some competitions, and under the code of points system at other events. It is hardly fair to the skaters not to have a consistent set of rules and judging standards that can be applied at all competitions.

The new system does not address the fundamental problem

Let's step back to consider the original problem the ISU was trying to address by proposing the code of points system.

Figure skating has been in a crisis because judges have been caught cheating and have admitted to cheating, and have gotten off with trivial punishments. The problem is cheating judges, not the scoring system.

Instead of addressing the problem of cheating judges by adopting a strong code of ethics, public accountability and inspection of the entire judging process, and a policy of lifetime bans for anyone caught trying to "fix" competition results, the ISU has merely adopted policies that will make it impossible for the public ever again to detect instances of cheating. Instead of making an effort to ban cheating judges for good, the ISU has effectively acknowledged and institutionalized cheating through their system of random selection of judges. Instead of subjecting judges' actions to tighter scrutiny, the ISU has eliminated event review meetings and made it impossible for the technical committees to scrutinize and evaluate judges' performance.

Introducing a lot of numbers and gee-whiz computer equipment to the judging process does not instill ethics in judges or prevent them from cheating. Making the judging process secret and incomprehensibly complex does not instill public confidence in the ethics of the judges or integrity of the sport. Adopting radical changes in skating standards and the balance between art and sport without open debate and consensus in the skating community does not instill fairness to the athletes.

Under Cinquanta's direction, the ISU has spent a great deal of money developing and promoting the code of points system. For what reason? Perhaps Cinquanta, as a former speed skater, would like to see everything in figure skating reduced to a single absolute number, like the single number that represents a speed skater's time.

But the code of points system does not change the fundamentally subjective nature of judging figure skating. Everything about the code of points system is subjective: the relative point values which have been assigned to different elements and program components, the levels assigned to elements by the "caller", the grade of execution assigned by the judges. Assigning point values to everything does not make the judging any less subjective or more accurate. It is a fundamentally futile endeavor, and it is time to recognize that and put this "project, not rule" aside.

Instead, skating must concentrate directly on solving the problem of credibility and public trust in the sport. We must see the organizations that govern skating follow their own rules, and an end to threats and intimidation. We must see individuals who are guilty of ethical violations banned permanently from the sport. We must see the entire process opened up to public scrutiny, and do away with secrecy. We must see figure skating being run by figure skaters, not by speed skaters or computer programmers. We must see the interests of the athletes given priority at every step of the way.

SkateWeb
 Home © 1994-2010 SkateWeb