Am I Being Dudded At Supercoach or What Does Champion Data Know About Aussie Rules Anyway?

One of the constant frustrations for my AFL Herald-Sun Supercoach Team has been watching a player rack up a bucketload of possessions on the weekend only to have them score pitifully for my fantasy team come Monday morning. For those unfamiliar with the great time-waster that is AFL Supercoach, you pick a team of 22 players and they score points for you based on their effectiveness. However, the actual formula for a player’s score is a mystery; instead, you are simply told that it is based on a formula devised by a mob called Champion Data. A search of the Champion Data website reveals a few further details – effective long kicks score highly, miscued kicks score badly – but the precise formula remains a trade secret.

This raises the question of whether Champion Data are rewarding or penalizing players in a fair and accurate manner. They claim that their formula has been devised using ‘research into winning and losing factors in AFL games’. This could mean that, the higher your Champion Data score, the more likely your team’s score will be higher relative to the other team (i.e. your team’s percentage will be higher) or that the higher your score, the more likely it is that your team will win, whether it be by one point or 100 points. I’m going to assume the former explanation is true, since it seems to make more sense. But how well does it work in practice?

Not that surprisingly, it tends to work pretty well. The figure below compares the actual percentage of each AFL team over the first 21 rounds of the 2007 season to their estimated percentage using their cumulative Champion Data scores. Apart from West Coast, the results are all pretty close.

Team Percentages and Cumulative Champion Data Scores Over 2007 AFL Season


Importantly, the Champion Data scores appear to perform better at predicting a team’s success than simply looking at that team’s number of possessions. The root mean squared error using the Champion Data scores (which is basically just a method of summarizing the difference between a team’s actual percentage and its predicted percentage) is about half of that which you would get if you used the number of possessions. Incidentally, the formula for the Dream Team competition on the AFL website doesn’t do much better at predicting a team’s success than possessions do.

Formula	Root Mean Squared Error
Champion Data	8.1
Possessions	15.7
AFL Dream Team	15.1

Readers of my post Win Score and the Productivity of Basketball Players will know that post also talked about a formula that did pretty well at predicting the success of a (basketball) team, but that I had reservations about how well it did at attributing this success to particular players. So how well does the Champion Data formula do in this respect? My guess is that similar types of problems apply, namely how do you account for the value of defenders who aim to prevent other players from collecting possessions rather than gathering possessions themselves, and how much credit should be given to the player who ultimately puts the score on the board? But I’m fairly satisfied that the Champion Data formula does a better job than simply looking at the raw numbers. On the other hand, if my Supercoach team loses its Grand Final this weekend, well…