Counterfactual reasoning and learning systems: the example of computational advertising

By L. Eon Bottou, Jan Peters, Joaquin Quiñonero Candela, Denis Xavier Charles, David Maxwell Chickering, Elon Portugaly, Dipankar Ray, Patrice Y. Simard, Edward Snelson. 2012

859 citations recorded by the source corpus.

This work shows how to leverage causal inference to understand the behavior of complex learning systems interacting with their environment and predict the consequences of changes to the system. Such predictions allow both humans and algorithms to select the changes that would have improved the system performance. This work is illustrated by experiments on the ad placement system associated with the Bing search engine.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Can Neural Machine Translation be Improved with User Feedback? (2018, background): “They show that variance reduction techniques used in counterfactual bandit learning (Dudı́k et al., 2011; Bottou et al., 2013) and off-policy reinforcement learning (Precup et al.”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, methodology): “…function approximation, have been presented under the name of “off-policy” or “counterfactual” learning (see Precup et al. (2000), Precup et al. (2001), Bottou et al. (2013), Swaminathan and Joachims (2015a), Swaminathan and Joachims (2015b), Jiang and Li (2016), Thomas and Brunskill (2016), inter…”
Dialogue Learning With Human-In-The-Loop (2016, positive): “Our experiments show that our model and base algorithms are very robust to the choice of batch size, alleviating the need for correction terms in the learning algorithm (Bottou et al., 2013).”
Toward Causal Representation Learning (2021, background): “The latter contains the mechanisms giving rise to the observed statistical dependences and allows to model distribution shifts through the notion of interventions [35], [180], [183], [188], [220], [237].”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “…offline data has included studies on optimizing newspaper article click-through-rates (Strehl et al., 2010; Garcin et al., 2014), advertisement ranking on search pages (Bottou et al., 2013), and personalized ad recommendation for digital marketing (Theocharous et al., 2015; Thomas et al., 2017).”
Deep Kalman Filters (2015, background): “For insightful work on counterfactual inference, in the context of a complex machine-learning and ad-placement system, see Bottou et al. (2013).”
How algorithmic confounding in recommendation systems increases homogeneity and decreases utility (2017, methodology): “[8], formal causal inference techniques can assist in the design of deployed learning systems to avoid confounding.”
Towards Causal Representation Learning (2021, background): “to model distribution shifts through the notion of interventions [183, 237, 218, 34, 188, 181].”
Causally Correct Partial Models for Reinforcement Learning (2020, background): “…explored for off-policy policy evaluation (Sutton & Barto, 2018; Precup, 2000; Precup et al., 2001; Munos et al., 2016; Espeholt et al., 2018; Nachum et al., 2018) as well as for counter-factual policy evaluation (Buesing et al., 2019) and for counter-factual model evaluation (Bottou et al., 2013).”
Causality for Machine Learning (2019, background): “Jonas was happy to take up the challenge, contributing to the work of Bottou et al. (2013), an early use of causality to learn large scale interacting systems.”
Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search (2018, background): “5 R ELATED W ORK Bottou et al. (2013) provide an in-depth discussion of applying models to off-policy evaluation.”
Causal inference by using invariant prediction: identification and confidence intervals (2015, background): “Besides understanding structural relations, causal information may lead to improved performance in some prediction problems [Bottou et al., 2013, Sch¨olkopf et al., 2012].”
The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations (2019, background): “Thisisin particular the case for counterfactual example approaches that, based on counterfactual reasoning (see e.g. [Bottou et al. , 2013]), aim at answering the question: given a classiﬁer and an observation, how is the prediction altered when the observation changes?”
Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (2015, background): “In robotics and business/marketing applications, for instance, it is often risky (thus expensive) to run a policy without an estimate of the policy’s quality (Li et al., 2011; Bottou et al., 2013; Thomas et al., 2015a).”
Counterfactual Estimation and Optimization of Click Metrics for Search Engines (2014, background): “ It is the focus of this paper to study how unbiasded oine policy evaluation can be done when optimizing search engines. 3. UNBIASED OFFLINE EVALUATION 3.1 Basic Techniques As is observed previously [5], oine policy evaluation can be interpreted as a causal inference problem, an important research t”