Responsible editor: R. Bayardo.

By Ron Kohavi, R. Longbotham, Dan Sommerfield, Randal M. Henne

774 citations recorded by the source corpus.

The web provides an unprecedented opportunity to evaluate ideas quickly using controlled experiments, also called randomized experiments, A/B tests (and their generalizations), split tests, Control/Treatment tests, MultiVariable Tests (MVT) and parallel flights. Controlled experiments embody the best scientific design for establishing a causal relationship between changes and their influence on user-observable behavior. We provide a practical guide to conducting online experiments, where end-users can help guide the development of features. Our experience indicates that significant learning and return-on-investment (ROI) are seen when development teams listen to their customers, not to the Highest Paid Person’s Opinion (HiPPO). We provide several examples of controlled experiments with surprising results. We review the important ingredients of running controlled experiments, and discuss their limitations (both technical and organizational). We focus on several areas that are critical to experimentation, including statistical power, sample size, and techniques for variance reduction. We describe common architectures for experimentation systems and analyze their advantages and disadvantages. We evaluate randomization and hashing techniques, which we show are not as simple in practice as is often assumed. Controlled experiments typically generate large amounts of data, which can be analyzed using data mining techniques to gain deeper understanding of the factors influencing the outcome of interest, leading to new hypotheses and creating a virtuous cycle of improvements. Organizations that embrace controlled experiments with clear evaluation criteria can evolve their systems with automated optimizations and real-time analyses. Based on our extensive practical experience with multiple systems and organizations, we share key lessons that will help practitioners in running trustworthy controlled experiments.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Statistics and Probability, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Experimental and Cognitive Psychology

In Statistics and Probability

In Evaluation & Benchmarks

Citations in the corpus

Counterfactual reasoning and learning systems: the example of computational advertising (2012, background): “The basic idea is to randomly split the users into treatment and control groups (Kohavi et al., 2008).”
Counterfactual Estimation and Optimization of Click Metrics for Search Engines (2014, background): “Controlled experiments have proved very successful in practice ( e.g. , [18]), allowing engineering and business decisions to be made in a data-driven manner.”
Off-policy evaluation for slate recommendation (2016, neutral): “While these approaches require less data, they can have large bias, and their use in practice requires an expensive trial-and-error cycle involving weeks-long A/B tests to develop new policies [20].”
Causal discovery in social media using quasi-experimental designs (2010, methodology): “The first is online experimentation, which is widely used [22, 14], and is an ideal tool for causal discovery.”
Learning to Learn without Gradient Descent by Gradient Descent (2016, methodology): “We are therefore considering the finite horizon setting that is popular in AB tests (Kohavi et al., 2009; Scott, 2010) and is often studied under the umbrella of best arm identification in the bandits literature (Bubeck et al., 2009; Gabillon et al., 2012).”
Strength in Numbers: How Does Data-Driven Decisionmaking Affect Firm Performance? (2011, background): “As a Microsoft researcher memorably put it, objective, fine-grained data are now replacing HiPPOs (Highest Paid Person’s Opinions) as the basis for decision-making at more and more companies (Kohavi et al., 2009).”
Video stream quality impacts viewer behavior: inferring causality using quasi-experimental designs (2012, background): “However, there are other domains were a controlled experiment can and are performed, example, A|B testing of web page layouts [11].”
A few useful things to know about machine learning (2012, neutral): “Second, if you can obtain experimental data (for example by randomly assigning visitors to different versions of a Web site), then by all means do so [14].”
Better never than late: meeting deadlines in datacenter networks (2011, background): “Customer studies guide the time in which users need to be responded to [14], and after accounting for wide-area network and rendering delays, applications typically have an SLA of 200-300ms to complete their operation and ship out the”
Evaluating Recommendation Systems (2011, background): “For this reason, many real world systems employ an online testing system [32], where multiple algorithms can be compared.”
A Survey of Accuracy Evaluation Metrics of Recommendation Tasks (2009, background): “Furthermore, one can view a predicted high rating as a recommendation to use the item, and a predicted low rating as a recomm ndation to avoid the item.”
Counterfactual Evaluation and Learning for Search, Recommendation and Ad Placement (2016, background): “Unlike deploying new systems in weeks-long A/B tests to get unbiased performance estimates [10], counterfactual estimators can provide unbiased estimates from existing logs without delay.”