Counterfactual Risk Minimization

By Adith Swaminathan, Thorsten Joachims. 2015

172 citations recorded by the source corpus.

We develop a learning principle and an efficient algorithm for batch learning from logged bandit feedback. This learning setting is ubiquitous in online systems (e.g., ad placement, web search, recommendation), where an algorithm makes a prediction (e.g., ad ranking) for a given input (e.g., query) and observes bandit feedback (e.g., user clicks on presented ads). We first address the counterfactual nature of the learning problem through propensity scoring. Next, we derive generalization error bounds that account for the variance of the propensity-weighted empirical risk estimator. These constructive bounds give rise to the Counterfactual Risk Minimization (CRM) principle. Using the CRM principle, we derive a new learning algorithm -- Policy Optimizer for Exponential Models (POEM) -- for structured output prediction. We evaluate POEM on several multi-label classification problems and verify that its empirical performance supports the theory.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Counterfactual Normalization: Proactively Addressing Dataset Shift and Improving Reliability Using Causal Mechanisms (2018, background): “Specifically, they optimize the counterfactual risk to explicitly account for variations in policy between train and test environments (e.g., Swaminathan and Joachims (2015); Schulam and Saria (2017)).”
Policy Evaluation and Optimization with Continuous Treatments (2018, methodology): “We tested discrete off-policy optimization (POEM and NORM-POEM) with various uniformlyspaced discretizations or quantile-based discretizations of dosages, but the propensity scores are mostly zero or one and hamper the resulting optimization, illustrating the difficulty of finding appropriate discre”
A General Framework for Counterfactual Learning-to-Rank (2018, background): “nt (SGD) is not directly feasible at the level of individual documents. In the case of DCG, since the rank weighting function is concave, one possible workaround is a Majorization-Minimization scheme [22] (akin to CCP): upper bound the loss function with a linear Taylor approximation at the current ”
A Survey on Causal Inference (2020, methodology): “Later, various methods [8, 16, 64, 76, 134, 135, 137, 155] are proposed to improve those two methods with different settings.”
Reliable Decision Support using Counterfactual Models (2017, background): “In RL, a related problem known as off-policy evaluation also uses retrospective observational data (see e.g., Dudík et al. 2011, Swaminathan and Joachims 2015, Jiang and Li 2016, Păduraru et al. 2012, Doroudi et al. 2017).”
Policy Evaluation with Latent Confounders via Optimal Balance (2019, background): “Since these weights can be extreme, both normalization [2, 31, 41] and clipping [10, 12, 40] are often employed.”
MultiVerse: Causal Reasoning using Importance Sampling in Probabilistic Programming (2019, background): “…(Constantinou et al., 2016; Schulam and Saria, 2017; Richens et al., 2019; Oberst and Sontag, 2019), advertisement and search (Bottou et al., 2013; Swaminathan and Joachims, 2015; Li et al., 2015; Gilotte et al., 2018), translation (Lawrence et al., 2017), and reinforcement learning (Foerster et…”
Learning to Personalize from Observational Data (2016, background): “POEM and NPOEM [16, 17] solve the problem of learning from logged bandit feedback, assuming access to the logging policy that generated the data.”
Modeling User Exposure in Recommendation (2015, background): “, log data) is therefore of important practical interest [3, 13, 28].”
Efficient iterative policy optimization (2016, methodology): “The need to perform rollouts after each update was alleviated by using importance sampling policy gradient [Swaminathan and Joachims, 2015].”
Learning Optimal Policies from Observational Data (2018, methodology): “• IPS is POEM without variance regularization.”
Causal embeddings for recommendation (2017, methodology): “In a series of papers (Swaminathan and Joachims 2015b; Swaminathan and Joachims 2015a), the authors propose the Counterfactual Risk Minimization (CRM) principle along with a new learning setup Batch Learning from Bandit Feed-back (BLBF) that aims to answer the counterfactual question of: ”How will…”
Confounding-Robust Policy Improvement (2018, methodology): “These fall under regression-based strategies [21] or reweighting-based strategies [3, 12, 13, 28], or doubly robust combinations thereof [6, 30].”