Statistical Analysis of Program Components Marks

by Sandra Loosemore
November, 2003

Introduction

The ISU's propaganda for the Code of Points claims that a major advantage of this system is that it will lead to "better consistency among judges" because the program components are based on "precise criteria". In other words, if the system is working the way the ISU says it should, we should expect to see the judges' marks for each component for a particular performance in close agreement, because their marks are now supposed to be based on "precise criteria" and have meaning in and of themselves instead of being placeholders for an ordinal, as in the 6.0 system. On the other hand, because the different program components each have their own "precise criteria", we would also expect to see significant variation in the marks for different components in the same performance, since skaters are unlikely to be equally good (or bad) at each component. For example, we might see skaters who have superior Skating Skills but a program with poor Choreography and simplistic Transitions.

Alas, even cursory examination of the results from the Grand Prix events where the Code of Points has been used shows that the expected judging pattern for the program components is not appearing. Instead, we are seeing comparatively little variation in the marks for the different program components given to each performance.

This article reports on a statistical analysis of the judging data from these competitions. The purpose of this analysis is to quantify and compare, for each performance,

• the variation between different judges' marks for each program component; and

• the variation between each judges' marks for different program components.

Statistical background

If the values are highly correlated and concentrated around the same point in the range, that corresponds to a steep bell curve like the one on the left. If, on the other hand, the data points are more "spread out" over a larger range, that corresponds to a shallower bell curve, like the one on the right.

Mathematically, the amount of variation in a set of data is called the standard deviation. You don't really need to understand how standard deviation is computed, but a smaller standard deviation means that the data is more closely correlated than a larger standard deviation does. More precisely, if the standard deviation is 1, then about 68% of the samples lie within plus or minus 1 of the mean, and about 95% lie with plus or minus 2 of the mean.

In terms of how this applies to the judging of the program components, if the new judging system is working the way the ISU claims it should, we should see that the standard deviation of the judges' marks for the same component is small, corresponding to a steep bell curve like the one on the left. On the other hand, since the different program components are not supposed to be highly correlated to one another, the standard deviation between the marks given to different program components by an individual judge should be larger, like the spread-out bell curve on the right.

The purpose of the statistical analysis being reported here is to test the ISU's hypothesis, by computing standard deviations in the program components marks for each performance at the Grand Prix events. If you are not interested in the technical details of exactly what was computed and how, you can skip straight to the Commentary section at the end for the discussion of the results.

Methodology

The program does not examine the correlation between the program components scores and the technical elements scores, or try to account for the random selection of judges and dropping of high and low marks which are also part of the Code of Points system.

Results

Commentary

In other words, instead of representing performance aspects that the judges can accurately and consistently differentiate and mark according to precise criteria, different judges' marks for the same component vary more than the marks from individual judges for components which are supposed to reflect completely different performance aspects. Not only that, but this counter-intuitive pattern holds for all but one performance out of the hundreds which have been evaluated at these competitions.

What does this say about the Code of Points judging system, and the way the judges are applying it?

• Having five program components marks offers no more accuracy in the judging than a single presentation mark, if the judges are unable to distinguish the criteria for the separate marks to the extent that it is greater than the "noise" of the variation between marks from different judges.

  A statistical analysis cannot pinpoint why the judges are failing to distinguish between the different program components criteria. Some possibilities are:

  • The judges lack training in applying the new rules (why did the ISU start using the system to determine actual competition results before the judges could be adequately trained?)

  • The program components criteria are too complicated for even trained judges to apply in real-time (why did the ISU start using the system to determine actual competition results before testing could reveal this flaw?)

  • The judges are engaging in "protocol judging" or otherwise deliberately attempting to manipulate the results (why did the ISU think that a system with no public accountability of the judges for their marks would fix this long-standing problem?)

• Many judges seem not to be taking the required 0.5 deduction from the Performance/Execution mark for each fall or other -3 error on a technical element. For example, in the men's free skate at Skate Canada, Sandhu, Honda, Takahashi, Jeannette, and Timchenko each had at least two -3 errors yet nearly all the judges awarded them Performance/Execution marks as high as, or even higher than, their marks for the other four components; while there was no pattern of rewarding skaters who did not fall with higher marks for Performance/Execution than in the other categories.

• Carrying out the computations to two decimal places conveys a misleading impression of accuracy in the marks because of the "noise" or variation between judges, even if random selection is not used.

• The variation between different judges' marks for the same component is significant enough that the random selection of judges used by the Code of Points system is likely to affect the outcome of competitions. Consider a variation of only half a point in each of the five program components; when multiplied by five components and the 2.0 factor for the men's free skate, that can translate into a 5-point difference, and there have already been many placements decided by less than than amount.

• Because the code of points system discards the high and low marks for each program component, marks from individual judges who do attempt to apply the stated judging standards using the full range of marks are more likely to be ignored in the overall computation of results, than those from judges who "protocol judge" and give marks in the middle of the range. The trimmed mean method assumes that the marks are incorrect simply because they are at the end of the range, when in fact the extreme marks may be more accurate and correct than those near the middle.

• The process the ISU has adopted for evaluating judges' performance may actually be contributing to the apparent "protocol judging" and the unwillingness of judges to use the full range of marks for the individual program components. Instead of being called upon to defend their marks based on the actual skating at the event, judges are liable to be disciplined only for giving marks that are out of line with the rest of the panel. Therefore they have more motivation to mark according to what they think the other judges are likely to do, than by what they think about the skaters' performances.

SkateWeb

Home	© 1994-2010 SkateWeb