6 Comments

Summary:

Bayes Theorem calculates the probability of possible outcomes based on previous events. Coupled with cloud computing and big data, it does more than tell you whether to choose door number one, two or three; it’s become the foundation for predictive analytics.

The goat-door problem AKA the Monty Hall Problem
photo: Susan Jarema

If contestants on the game show Let’s Make a Deal knew Bayes’ Theorem, they’d be much more likely to win the big ticket prize than head home with the undesirable items the show referred to as “Zonks.”

The show used what came to be known as the Monty Hall Problem, a probability puzzle named after the original host. It works like this: You choose between three doors. Behind one is a car and the other two are Zonks. You pick a door – say, door number one – and the host, who knows where the prize is, opens another door – say, door number three – which has a goat. He then asks if you want to switch doors. Most contestants assume that since they have two equivalent options, they have a 50/50 shot of winning, and it doesn’t matter whether or not they switch doors. Makes sense, right?

A data scientist, on the other hand, would use Bayes’ Theorem to calculate the probability of winning the car, instead of relying on their intuition. And they’d have a much higher chance of winning. The correct answer, it turns out, is to switch the door.

Thomas Bayes, born in 1703, was an English mathematician, statistician and Presbyterian minister. The work that made him famous – Bayes Theorem – wasn’t published until two years after his death. This has become the preferred method for calculating the probability of an event that is dependent on other events preceding it. A multitude of modern techniques used by data-scientists, e.g., naive Bayes, Bayesian estimation, Bayesian Networks, etc., all derive from the original Bayes Theorem.

One theorem, many impressive applications

Whether you realize or not, Bayes Theorem is used in many of the consumer technologies we use daily. It allows us to make predictions on everything from the weather, to fantasy football and presidential elections. It powers Amazon and LinkedIn’s recommendations and Google’s search products. It even keeps spam out of your inbox.

The Monty Hall Problem is also at the core of business. Just as a Let’s Make a Deal contestant can better their chance of winning by using probability theory, so can businesses. As enterprises move away from gut-based decision making to relying on data science for applications in risk management, marketing, sales, hiring, retaining customers and pricing correctly, it’s no surprise that Bayes Theorem has started to make a broader impact on the economy. Where people often guess the probability of events incorrectly, Bayes Theorem does not.

The Theorem considers “Priors” (also called “signals”) – events that we have pre-existing knowledge on – and “Posteriors” – events that we want to predict. As more data on Priors has become available, one can use cloud computing to calculate the probability of even very complex events using Bayes Theorem. Modern predictive analytics is powered by two major trends – big data and cloud computing – and Bayes Theorem is the link that connects the two.

Why Bayes’ is even more important today

When businesses use Bayes’ Theorem to solve a complex challenge, the accuracy of the prediction is limited by the quantity and accuracy of Signals. While machine learning allows models, which often have a learning curve, to improve over time, the key to better predictions is higher quality data on Signals. This is where cloud-based providers of data and insight will have the largest impact since they can harvest Signals at a scale that’s hard to replicate at the individual enterprise level.

They do what Nate Silver calls the “Bayesian convergence” – dispel myths and opposing opinions as evidence of the most likely outcome is uncovered. This is Bayes’ greatest gift to enterprise technology.

Bayes Theorem isn’t shiny or new. In fact, it is taught in most high schools across the world and is actually a very logical way of thinking about business decisions. It’s been around for a long time – 251 years to be exact. But the value that Thomas Bayes has brought to the enterprise has become considerably more apparent alongside the progression of Big Data. Every business is capable of using these methodologies for decision making. Don’t let yours operate like a Let’s Make a Deal contestant.

Shashi Upadhyay is co-founder and CEO at Lattice, a company that offers predictive applications that help companies market and sell more intelligently. Follow him at @shashiSFGigaom’s Structure:Data event, held March 19-20 in New York City will examine other tools companies are using to analyze data.

(This post has been edited to reflect that probability theory can improve the odds of solving the Monty Hall Problem.)

By Shashi Upadhyay, Lattice

You're subscribed! If you like, you can update your settings

Related stories

  1. As one who has taught Bayesian statistics and decision theory for two decades, let me make a comment here.

    The description of the “Monty Hall Problem” is correct, and it is true that if that is the situation, it pays to switch doors.

    However, Monty Hall himself has said that this is not the way the game was actually set up. In the problem setup, which I call “Regular Monty”, Monty *always* opens a door that is not the door that the contestant has chosen, and the door he opens *never* has the prize.

    But in the actual TV show, Monty did not always open a door, and if he did it could well be the door that the contestant chose, or it could be the one that had the prize. Sometimes, Monty would even give the contestant a chance to switch without opening a door. So the conditions of the puzzle are not satisfied.

    In my courses, I posited alternative versions of the Monty Hall problem: “Angelic Monty” opens the door you’ve chosen if you chose the right door, and only gives you a chance to switch if you’ve chosen the wrong door. In this case, if you knew that you were facing Angelic Monty, you ought to switch if offered a chance.

    But “Monty From Hell” always opens the door with the prize if you chose the *wrong* door, showing you that you’ve lost, and if you’ve chosen the *right* door he opens one of the two doors without the prize and offers you a chance to switch. If you knew that you were facing “Monty From Hell”, you should *never* switch if offered a chance.

    “Ignorant Monty” doesn’t know where the prize is. He always opens a door at random, which sometimes has the prize (in which case you’ve lost) and sometimes does not, and if the door he opens does not have the prize, he offers you a chance to switch. In this version, there is no advantage to switching.

    “Mixture Monty” flips a fair coin out of sight of the contestant before the contestant comes on. If the coin comes up heads, he behaves as “Angelic Monty”; if it comes up tails, he behaves as “Monty From Hell”. In this case, if he opens a door and it does not have the prize, you should switch; the probabilities are the same as for “Regular Monty” as described in the article, even though the analysis is different.

    So the answer (switch or don’t switch) depends on the rules that are in effect. The original Monty Hall as played on TV did not have a fixed set of rules, so the analysis doesn’t have a fixed answer.

    Share
    1. Shashi Upadhyay Tuesday, February 18, 2014

      Thank you for your detailed comment. Since the article is aimed at a general audience with minimal familiarity with probability theory, I used the textbook version of the Monty Hall problem.

      Shashi

      Share
    2. I’ll give you $100 if you have a hard-boiled egg in your pocket right now!

      Share
  2. The phrase “using probability rather than logic” seems misguided to me … probability is most definitely logical, even though the logic can be fairly complicated (as the comment from “bayesrules” shows) … and, of course, the article itself later correctly notes that “Bayes Theorem … is actually a very logical way of thinking”

    Share
    1. Shashi Upadhyay Tuesday, February 18, 2014

      You are absolutely correct – the sentence is somewhat misleading. What I meant to say was that situations with incomplete information can’t be predicted using logic alone, and you have to use probabilistic concepts instead.

      Thank you for the clarification.

      Shashi

      Share
  3. Afra Zomorodian 1998 report on the same subject

    http://www.cs.dartmouth.edu/~afra/goodies/monty.pdf

    Share

Comments have been disabled for this post