What "American Idol" Can Teach Us About Stats

Stats are everywhere these days. From ballot measures to the economy to health issues to baseball — there are statistical points and counterpoints enough to confuse almost any topic. And often, more often than you would guess, the way we measure something significantly influences the final results.

Indeed, how you count something is as important as what you count. For an excellent example of this in action, look no further than “American Idol.”

To be clear, I am not alleging ballot tampering or conspiracy against certain contestants. Measuring popularity is challenging. But the vote-counting process for “American Idol” has systemic limitations. The results of which are being billed as a precise measure, when in fact, statistically they are not.

This revelation came to me during a recent episode, when I was listening to my 14-year-old daughter lament over how front-runner Adam Lambert was only 1 million votes ahead of the next competitor going into the season 8 finals after over 88 million votes were cast. She was frustrated because she repeatedly hit a busy signal for Adam’s vote line, but she got through more easily for other contestants.

The contest, of course, is not measuring intended votes for each contestant. Rather it’s only counting the votes that are actually make it through the clogged phone lines. A high volume of should-count-but-don’t votes overwhelms the system.

Think of it like a funnel, in which there are different amounts of, say, jelly beans are at the top but only a fixed number of jelly beans make it through to the bottom. In effect, the system is biased to equalize contestant voting to the maximum capacity of the phone lines through which the calls are made. That means that as the number of contestants gets smaller and the vote count gets higher, mathematically, the system biases the results to be close.

For example, say contestant A was extremely popular, and had 10 million calls per minute placed with the aim of voting for him/her. Contestant B was much less popular, receiving just 1 million calls per minute. If the call center could only process and tabulate 1 million calls/minute for each line, then the contestants would have an equal number of votes — contestant A would simply generate an additional 9 million busy signals or rejected calls. Not exactly an accurate representation of their respective popularity.

The system constraints create a statistical artifact of closeness. Further, as the would-be voters for the popular candidate encounter more busy signals, they will be less likely to continue to vote, actually improving the odds of the less popular candidate. By restricting the eligible voting period to two hours following the telecast, it further constrains the system, forcing more and more busy signals — or non-counted votes — into the equation.

A more fair way to count the votes would be to have several phone lines, all of which would give the caller the choice to vote for any of the candidates. Or perhaps an online voting system might be better suited to counting the results, especially since the contest allows multiple votes per person.

Making the results of “American Idol” voting closer than they appear might have been intentional, a bid to boost suspense and ratings. One way or another, the system does not accurately represent voters’ intention. It offers instead a rough estimate with a systemic error — an error that can mean popular contestants can get rejected in favor of less popular ones. It also makes for a terrific lesson in statistics.

Peter Daboll is the CEO of Bunchball. He previously held the Chief of Insights post at Yahoo and prior to that, was president and CEO of comScore Media Metrix.