A/B tests to determine which features or designs work best is a standard tool of the trade for any data scientist working on a web or mobile application. On Thursday, Facebook released a portion of its A/B testing code, called PlanOut, that’s focused on helping data scientists easily build and manage experiments while ensuring the results are accurate.
A blog post, written by Facebook data scientists Eytan Bakshy, Dean Eckles and Michael Bernstein, describes the importance of running such experiments correctly:
“At Facebook, we run over a thousand experiments each day. While many of these experiments are designed to optimize specific outcomes, others aim to inform long-term design decisions. And because we run so many experiments, we need reliable ways of routinizing experimentation. … Many online experiments are implemented by engineers who are not trained statisticians. While experiments are often simple to analyze when done correctly, it can be surprisingly easy to make mistakes in their design, implementation, logging, and analysis.”
For more on the importance of A/B tests and generally measuring every aspect of the web user experience, check out this panel from our recent Structure Data conference featuring data scientists from Airbnb, LinkedIn and Uber.