# What "American Idol" Can Teach Us About Stats

Summary:

Stats are everywhere these days. From ballot measures to the economy to health issues to baseball — there are statistical points and counterpoints enough to confuse almost any topic. And often, more often than you would guess, the way we measure something significantly influences the final […]

Stats are everywhere these days. From ballot measures to the economy to health issues to baseball — there are statistical points and counterpoints enough to confuse almost any topic. And often, more often than you would guess, the way we measure something significantly influences the final results.

Indeed, how you count something is as important as what you count. For an excellent example of this in action, look no further than “American Idol.”

To be clear, I am not alleging ballot tampering or conspiracy against certain contestants. Measuring popularity is challenging. But the vote-counting process for “American Idol” has systemic limitations. The results of which are being billed as a precise measure, when in fact, statistically they are not.

This revelation came to me during a recent episode, when I was listening to my 14-year-old daughter lament over how front-runner Adam Lambert was only 1 million votes ahead of the next competitor going into the season 8 finals after over 88 million votes were cast. She was frustrated because she repeatedly hit a busy signal for Adam’s vote line, but she got through more easily for other contestants.

The contest, of course, is not measuring intended votes for each contestant. Rather it’s only counting the votes that are actually make it through the clogged phone lines. A high volume of should-count-but-don’t votes overwhelms the system.

Think of it like a funnel, in which there are different amounts of, say, jelly beans are at the top but only a fixed number of jelly beans make it through to the bottom. In effect, the system is biased to equalize contestant voting to the maximum capacity of the phone lines through which the calls are made. That means that as the number of contestants gets smaller and the vote count gets higher, mathematically, the system biases the results to be close.

For example, say contestant A was extremely popular, and had 10 million calls per minute placed with the aim of voting for him/her. Contestant B was much less popular, receiving just 1 million calls per minute. If the call center could only process and tabulate 1 million calls/minute for each line, then the contestants would have an equal number of votes — contestant A would simply generate an additional 9 million busy signals or rejected calls. Not exactly an accurate representation of their respective popularity.

The system constraints create a statistical artifact of closeness. Further, as the would-be voters for the popular candidate encounter more busy signals, they will be less likely to continue to vote, actually improving the odds of the less popular candidate. By restricting the eligible voting period to two hours following the telecast, it further constrains the system, forcing more and more busy signals — or non-counted votes — into the equation.

A more fair way to count the votes would be to have several phone lines, all of which would give the caller the choice to vote for any of the candidates. Or perhaps an online voting system might be better suited to counting the results, especially since the contest allows multiple votes per person.

Making the results of “American Idol” voting closer than they appear might have been intentional, a bid to boost suspense and ratings. One way or another, the system does not accurately represent voters’ intention. It offers instead a rough estimate with a systemic error — an error that can mean popular contestants can get rejected in favor of less popular ones. It also makes for a terrific lesson in statistics.

Peter Daboll is the CEO of Bunchball. He previously held the Chief of Insights post at Yahoo and prior to that, was president and CEO of comScore Media Metrix.

1. Peter, great post but I hate you for enticing me to wade into any discussion pertaining to American Idol.

Not to split hairs here, but much of American Idol’s voting is via SMS … no jellybean funnel operating there.

And the physical lines associated with voting are likely pooled between all contestant DIDs meaning that votes for all contestants must pass through the same physical choke point together. This means that while there will be a high standard error, the resultant votes will still be somewhat representative of overall sentiment.

That said, I’m sure the State of Florida is evaluating AI’s voting schema for the next federal election.

Share
1. Thanks for your post, ianbell

Agreed on being dragged into an American Idol discussion, I blame my daughter for that.

The point on SMS is a good one, not sure what proportion of the votes are SMS vs. dial-in. My guess is that it is still small relative to the total. Further, even the SMS recording likely have some volume limits.

I don’t know if all calls funnel through a single choke point or not, we would need to see the routing schema as you suggest. It would certainly improve representation if they did.

But the point was simply that without understanding how all of this works, there is a suspicion in the result.

Share
1. One thing about the SMS voting is that I believe it is only available to cell phone users on the ATT network.

They did decide to open up multiple lines for each contestant once the field narrowed, most likely for the reasons you mentioned in the post. Though this only decreases the phenomenon, not eliminates it.

The capabilities of reporting technologies and their effect on end-result statistics is an interesting concept.

Share
2. “But the point was simply that without understanding how all of this works, there is a suspicion in the result.”

I expect that sort of ambiguity is exactly what the Producers rely upon. :)

Share
2. I am so excited for tonight, I can’t believe its almost over! Get a recap by watching this. OMG the auditions are hilarious, def click to see it and pass on to your idol friends.

http://tinyurl.com/idolrecap09

Share
3. Interesting post here Peter. It would be interesting to see what percentage of voters vote via SMS as opposed to telephones. Living out in the bay area probably would make me think everyone uses SMS to ianbell’s point above, but I could be wrong.

With that said, I would love for you to check out Predictify’s American Idol Prediction challenge with EW.com – http://www.predictify.com/idol. This is an online way to at least predict if you are right or not. Maybe one day they will get the online voting system down.

Share
4. If AI voting works the same way it did five years ago when I was at AT&T, congestion fundamentally occurs at two bottlenecks. The first is the interconnection between the originating local network (end office or tandem) and the AT&T “long-distance” network; the second in the AT&T network.

At the local/4E interconnect, the local network applies network management controls on the six-digit code (866-436). These are fairly coarse controls — they uniformly throttle all attempts to all numbers in the affected code. So, say, 1 in 10 calls to any 866-436 number will be processed by the originating end office or access tandem (or equivalent VoIP softswitch); the other nine in ten will get a fast busy. Sucks for anyone who runs a business with an 866-436 number, but results in uniform blocking of calls to all AI contestants. So if 10 million people attempt to vote for contestant one and 1 million attempt to vote for contestant two, the originating network will put through 1M calls for contestant 1 and 100k calls for contestant 2 (plus or minus some random variance, which at these numbers will be small).

Within the AT&T network, the engineers did a remarkable job in tuning things to get the blockage rate astonishingly low — if my memory serves, I remember numbers like 35M attempts with less than 1M blocked. Within the AT&T network, all AI calls route to the same announcement and televoting resource pool. The digits dialed are used to designate which announcement to play (“Thank you for voting for contestant 2″), but the circuits are pooled, so the probability of a given call being blocked due to a lack of announcement circuits is the same regardless of which number was dialed. Think of it as a single trunk group from the switch to a great big pooled announcement system, with enough announcements to play any announcement to any incoming trunk, rather than a separate trunk group for each announcement. Basically, if 1 in 35 calls are being blocked, 1 in 35 are being blocked for every contestant – so the contestant with 1M votes getting to the AT&T network would get 28,500 blocked and 971,500 counted, while the contestant with 100k votes getting through would get 2,850 blocked and 97,150 counted.

Bottom line, your statement that “the system does not accurately represent voters’ intention” is, well, wrong. But don’t feel bad – there were hundreds of posts arguing this on comp.dcom.telecom.tech back in the day…

Share
1. Thank you DG Lewis for the clarification. Those of us nerds who got sucked into following AI – I blame my Aunt – feel much better knowing that the voting is being handled professionally. Now let me get back to trying to vote for Kris Allen before the time is up!

Share
2. AEvangelista Friday, May 29 2009

I just happened upon this discussion while Googling.

DG, I appreciate your insight and expertise, but it doesn’t seem to jibe with reality.

For instance, Ruben vs Clay was 24 million votes split almost perfectly in half. That doesn’t seem realistic. It seems more like the system maxed out and gave each a max of 12 million, without regard for actual votes.

Also DialIdol.com, a geek who counts busy percentages from users of DialIdol software, has shown itself to be a reasonable indicator of contestant popularity, at times even being freakishly accurate. It seems from what you describe, DialIdol shouldn’t work at all. But it does.

I believe we just witnessed Ruben vs Clay again, with Adam vs Kris. A virtual tie in the dial vote. But this time the decision was left to the text voters.

Share
5. Interesting post (I raised the same issue last week, when I was unable to get through), and interesting discussion points. I guess in the big scheme of things, I don’t really care, either, but it does sort of feel to me like they could have a better voting system. I suspect that there is a small percentage of extremely avid people who vote continuously throughout the open voting period. I guess you could make the point that this shows which contestant has the most avid fan base, and therefore is the most likely to sell the most records. But I have to assume that a lot, if not most, folks are like me – they decide to vote for someone, try calling a few times, get a busy signal each time, and just give up (because it seems ridiculous to keep calling over and over for several hours). Sometimes I think perhaps the producers should only allow one vote per phone line, or something similar, so that at the end of the day, the vote truly represents the “American” idol, and not just the person with the most persistant supporters.

Share
6. Hmm interesting post, I’m sure the smses get stucks when there’s too many of them as well. So this tells fans to KEEP CALLING DON’T EVER GIVE UP!!

Share
7. I like this. *thumbs up*

I say it is an iconspiracy to get tweens and teens to stop calling and start texting. A much more lucrative endeavour for Job and his peeps.

Phonecalls are totally yesteryear, KWIM?

Share
8. According to folks at ATT it actually is possible to have dedicated call acceptance and rejection lower the exchange (866-436) level because of the sheer call volume. So it is possible that the calls are not pooled, and that is why their could be diferent odds of getting through by different number, as the writer suggests.

It is crazy how we are all discussing american idol. Some in some crazy way, the producers win again.

grrr

Share
1. I don’t believe local exchanges or access tandems have ten-digit 8YY network management controls, although it’s possible that they do — and if they do, they’d impose the controls uniformly on all the numbers in the 866-436-570X tens group. Within the AT&T “long distance” network they can control on a ten-digit level, but again, they’d apply uniform controls on all the AI numbers. Remember, the point of NM controls aren’t to improve the chances of calls to overloaded numbers getting through, the point is to keep these calls from swamping all the other traffic in the network.

And I do know for a fact that the vote tallying and announcement resources are pooled – the guy that did a lot of the tweaking for AI used to work for me.

Share
9. agree with the observation and reasoning (somewhat) but not the conclusion. You said “an error that can mean popular contestants can get rejected in favor of less popular ones”. I agree that the margin between #1 and #2 will be less than it should be but I can’t see how this will change the rankings [unless the voting pattern biases the way people vote in subsequent rounds]

Share
10. [...] GIGAOM Stats are everywhere these days. From ballot measures to the economy to health issues to baseball [...]

Share
11. [...] What “American Idol” Can Teach Us About Stats Stats are everywhere these days. From ballot measures to the economy to health issues to baseball — there are [...] [...]

Share
12. [...] What American Idol Can Teach Us About Stats [...]

Share
13. I was not surprised by the outcome. First of all there was about a million votes separating the 2 top contenders.Danny and Kris are very much alike in the type of music they sing. Adam is very talented, but his music appeals to a much smaller audience which is not as mainstream. I think the voting was fair as far as the number of votes but I don’t think the system works very well. After the 3rd person is eliminated, that singer’s style will carry over to the one most like him. I think the same thing happened in American Idol 2 when Kimberly Locke was voted off. The people that had voted for her then voted for Ruben. Adam will be just as popular and successful coming in 2nd as he would if he were 1st. He has an amazing talent.I think we should just let it go and let Kris enjoy it He seems to be apologizing for winning.

Share
14. Brenda Brockwell Wednesday, May 27 2009

I would like to know how many votes for each.

Share

Comments have been disabled for this post