Today’s xkcd is a work of subtle comic genius. The strip’s regular readers will all understand it as a reference to a debate about validity of FiveThirtyEight’s election-forecasting technique. For anyone else, read on.
For stat-heads like me, the most exciting part of Tuesday’s vote was not the outcome itself, because anyone who looked at the data seriously realized that President Obama was an overwhelming favorite for re-election. Rather, it was the resolution of a silly debate between the innumerate majority of the blogosphere* and media and a small number of data-based forecasters, most prominently Nate Silver of FiveThirtyEight. Tons of people called the election a “tossup”, and critiqued Silver for calling Obama the favorite. A lot of this exhibited a misunderstanding of statistics. A 90% chance of victory does not mean that Obama will win for sure. It means that in 9 out of 10 runs of the election we’d expect him to win, with the remaining 1 in 10 being due to last-second shifts in the electorate toward Romney compared with the polls.
So how did Silver do? Unfortunately I can’t find a way to permalink to the results, but as of 9:10 AM Central African Time on November 7th, MSNBC’s results have Silver correctly predicting every single state that has been called, and correctly forecasting the leader in the states yet to be decided. That includes Florida, where a naive state-level polling average predicted a Romney win – but Silver adjusted those polls to account for recent movement toward Obama nationally.
This is an absolutely dominating performance. It is a triumph for the use of data over hand-waving, and for statistics as a field. If anything, Silver’s track record is such that his predictions were too modest. For many swing states, he only predicted the victor had a barely-better-than-50% chance of winning. If he was correctly judging his margin of error, he’d have made a couple of mistakes tonight. He was hedging against the chance that the polls were biased but in fact these days polls are pretty damned accurate.
Tomorrow morning, I want to see virtually every political commentator and all those crazy dudes running websites forecasting Romney landslides to admit that they were wrong, and that analyzing the data and using it to form the best possible forecasts not only works, but works spectacularly well.
*There were also critiques by fairly smart people, pointing out Silver’s lack of transparency and arguing that using 3 significant digits for the probability of an Obama win is unreasonable. This post is not about them.