Online controlled experiments at large scale

By Ron Kohavi, Alex Deng, Brian Frasca, T. Walker, Ya Xu, Nils Pohlmann. 2013

432 citations recorded by the source corpus.

Subfields: Immunology, Statistics and Probability, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

In Evaluation & Benchmarks

Citations in the corpus

ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (2020, background): “This poses a challenging tradeoff since raising query response times by as little as 100ms is known to impact user experience and even measurably diminish revenue [18].”
Interpretable & Time-Budget-Constrained Contextualization for Re-Ranking (2020, background): “ining of linear rankers [34], and comparing the effectiveness and efﬁciency of various learning-to-rank algorithms [4]. In web search the speed of a response is crucial as determined by Kohavi et al. [19] in a large scale experiment, however, in some expert tasks, users are willing to wait longer fo”
Query Driven Algorithm Selection in Early Stage Retrieval (2017, neutral): “moretimes using a learning-to-rank algorithm to produce a •nal output set oft documents, where t k, and can be t ˝k in some con•gurations. E†ciency ma−ers at all stages of the process. Kohavi et al. [31] showed that every 100 ms boost in overall search speed increases revenueby0:6%atBing. So, evensm”
MergeRUCB: A Method for Large-Scale Online Ranker Evaluation (2015, background): “[21] report that on any given day over 200 concurrent experiments are being run at Bing, with users ending up in one of billions of possible variants of the site.”
Sequential estimation of quantiles with applications to A/B testing and best-arm identification (2019, neutral): “A/B testing, the use of randomized experiments to compare two or more versions of an online experience, is a widespread practice among internet firms (Kohavi et al., 2013).”
Online Evaluation for Information Retrieval (2016, background): “For instance, in a web search setting, it has been observed that even imperceptible changes in the time it takes for an IR system to return results can have large impact on user behavior [Kohavi et al., 2013].”
MergeDTS (2018, background): “But, in real-world online ranker evaluation problems, experiments involving hundreds or even thousands of rankers are commonplace [30].”
Copeland Dueling Bandits (2015, background): “For example, at Bing, 200 experiments are run concurrently on any given day [14], in which case the duration of the experiment needs to be longer than the age of the universe in nanoseconds before K log T becomes significant in comparison to K2.”