header

	in partnership with
Providing Outreach in Computer Science		Bringing Bayesian solutions to real-world risk problems

Probability: Fallacies, Myths and Puzzles: Main Page

The Schools League Tables Fallacy

Look at the following table showing the scores achieved by a set of schools (that we have made anonymous by using numbers rather than names). School 38 achieved a significantly higher score than the next best school, and its score is over 52% higher than the lowest ranked school, number 41.

If I told you that these were the available state schools in your borough and that, instead of your child being awarded a place in school 38, he/she was being sent to school 41. You would be pretty upset, wouldn't you? Now scroll down to below the table.

School Number	Score
38	175
43	164
44	163
25	158
31	158
47	158
11	155
23	155
48	155
40	153
7	151
30	151
6	150
9	149
33	149
19	148
10	147
12	147
32	147
2	146
27	146
42	146
28	145
35	145
49	145
45	144
46	143
1	142
18	142
22	141
26	141
4	140
14	140
29	140
39	139
8	138
5	136
17	136
34	136
3	134
24	133
36	131
37	131
15	130
21	130
16	128
13	120
20	116
41	115

In fact the numbers do not represent schools at all. They are simply the numbers used in the UK National Lottery (1 to 49). And each ‘score’ is the actual number of times that particular numbered ball had been drawn in the 1172 draws of the UK National lottery that had taken place up to 17 March 2007. So the real question is: Do you believe that 38 is a 'better' number than 41? Or, making the analogy with the schools league table even more accurate, do you believe the number 38 is more likely to be drawn next time than the number 41? (after all your interpretation of the schools league table is that if your child attends the school at the top he/she will get better grades than if he/she attends the school at the bottom).

The fact is that the scores are genuinely random. Although the ‘expected’ number of times any one ball should have been drawn is about 144 you can see that there is a wide variation above and below this number (even though that is still the average score). What many people fail to realise is that this kind of variation is inevitable. To see why look at this explanation. If you rolled a die 60 times you would almost certainly not get each of the 6 numbers coming up 10 times. You might get 16 threes and only 6 fours. That does that not make the number three better than the number four. The more times you roll the die, the closer in relative terms will be the frequencies of each number (specifically, if you roll the die n times the frequency of each number will get closer and closer to n divided by 6 as n gets bigger); but in absolute terms the frequencies will not be exactly the same. There will inevitably be some numbers with a higher count than others. And one number will be at the 'top' of the table while another will be 'bottom'.

I am not suggesting that all schools league tables are purely random like this. But, imagine that you had a set of genuinely equal schools and you ranked them according to a suitable criteria like average A-level results. Then in any given year you would expect to see a wide variation like the table above. And you would be completely wrong to assume that the school at the top was any better than the school at the bottom.

In reality, there are inherent quality factors that help determine where a school will come in a league table. But this does not disguise the fact that much of the variation in the results will be down to nothing more than pure and inevitable chance.

footer
Norman Fenton

Return to Main Page Making Sense of Probability: Fallacies, Myths and Puzzles