Offline A/B Testing for Recommender Systems

By Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, Simon Dollé. 2018

256 citations recorded by the source corpus.

Online A/B testing evaluates the impact of a new technology by running it in a real production environment and testing its performance on a subset of the users of the platform. It is a well-known practice to run a preliminary offline evaluation on historical data to iterate faster on new ideas, and to detect poor policies in order to avoid losing money or breaking the system. For such offline evaluations, we are interested in methods that can compute offline an estimate of the potential uplift of performance generated by a new technology. Offline performance can be measured using estimators known as counterfactual or off-policy estimators. Traditional counterfactual estimators, such as capped importance sampling or normalised importance sampling, exhibit unsatisfying bias-variance compromises when experimenting on personalized product recommendation systems. To overcome this issue, we model the bias incurred by these estimators rather than bound it in the worst case, which leads us to propose a new counterfactual estimator. We provide a benchmark of the different estimators showing their correlation with business metrics observed by running online A/B tests on a large-scale commercial recommender system.

Subfields: Information Systems, Management Science and Operations Research, Statistics and Probability, Recommender Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Management Science and Operations Research

In Statistics and Probability

In Recommender Systems

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Applications of offline RL for recommender systems include slate and whole-page optimization (Swaminathan et al., 2017), applying doubly robust estimation to estimate website visits Dudík et al. (2014), and A/B testing for click optimization (Gilotte et al., 2018).”
Benchmarks for Deep Off-Policy Evaluation (2021, background): “…theoretical (Thomas & Brunskill, 2016; Swaminathan & Joachims, 2015; Jiang & Li, 2015; Wang et al., 2017; Yang et al., 2020) and practical progress (Gilotte et al., 2018; Nie et al., 2019; Kalashnikov et al., 2018) on OPE algorithms has been made in a range of different domains, there are few…”
Unbiased Recommender Learning from Missing-Not-At-Random Implicit Feedback (2019, methodology): “In this subsection, we apply the propensity clipping technique [4, 6, 24] to our estimator and analyze the bias-variance trade-off of an estimator with clipping.”
MultiVerse: Causal Reasoning using Importance Sampling in Probabilistic Programming (2019, background): “…Richens et al., 2019; Oberst and Sontag, 2019), advertisement and search (Bottou et al., 2013; Swaminathan and Joachims, 2015; Li et al., 2015; Gilotte et al., 2018), translation (Lawrence et al., 2017), and reinforcement learning (Foerster et al., 2018; Forney et al., 2017; Buesing et al.,…”
Explore, exploit, and explain: personalizing explainable recommendations with bandits (2018, background): “To further address small propensities giving the objective high variance, a combination of normalized importance sample reweighting and propensity score capping is used [7].”