Overlapping experiment infrastructure: more, better, faster experimentation

By Diane Tang, Ashish Agarwal, Deirdre O'Brien, Mike Meyer. 2010

336 citations recorded by the source corpus.

Subfields: Information Systems, Information Systems and Management

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Information Systems and Management

Citations in the corpus

Counterfactual reasoning and learning systems: the example of computational advertising (2012, background): “Finally, although there are elegant ways to efficiently run overlapping controlled experiments on the same traffic (Tang et al., 2010), they are limited by the volume of traffic available for experimentation.”
Ad click prediction: a view from the trenches (2013, background): “(Evaluating models on portions of live traffic is an important, but more expensive, piece of evaluation; see, for example, [30].”
Optimal Testing in the Experiment-rich Regime (2018, background): “In modern A/B testing (e.g., for web applications), it is not uncommon to find organizations that run hundreds or even thousands of experiments at a time [Kaufman et al., 2017, Tang et al., 2010a, Kohavi et al., 2009, Bakshy et al., 2014].”
Understanding User Behavior Through Log Data and Analysis (2014, methodology): “There are a variety of ways to calculate the standard error when the observations are not independent— two common ones are the delta method (Wikipedia: Delta method ) and using “uniformity trials” (Tang et al., 2010 ).”
Online controlled experiments at large scale (2013, neutral): “Another key tool for prevention uses a configuration management system to automatically detect experiments trying to change the same configuration parameter prior to launch [3].”
The production of prediction: What does machine learning want? (2015, background): “In an academic report describing how Google sets up many experiments to run simultaneously on its search services, Google researchers Tang et al. (2010) report, ‘the more general problem of supporting massive experimentation applies to any company or entity that wants to gather empirical data to…”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, methodology): “One way to mitigate this is to use proper A/B testing protocols as described in Tang et al. (2010).”