Batch learning from logged bandit feedback through counterfactual risk minimization

By Adith Swaminathan, Thorsten Joachims. 2015

396 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Science Applications, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback (2018, methodology): “Our contribution is to modify the self-normalizing estimator (Kong, 1992; Precup et al., 2000; Swaminathan and Joachims, 2015b; Joachims et al., 2018) to be applicable to neural networks.”
Deep-Treat: Learning Optimal Personalized Treatments From Observational Data Using Neural Networks (2018, methodology): “Self normalizing estimator, a variant of IPS estimator, is also used to learn counterfactuals (Swaminathan and Joachims 2015b).”
Balanced Linear Contextual Bandits (2018, positive): “2014), (Swaminathan and Joachims 2015), (Jiang and Li 2016), (Thomas and Brunskill 2016), (Athey and Wager 2017), (Kallus 2017), (Wang, Agarwal, and Dudı́k 2017), (Deshpande et al.”
Estimating Counterfactual Treatment Outcomes over Time Through Adversarially Balanced Representations (2020, background): “Learning optimal policies: A related problem to ours involves learning the optimal treatment policies from logged data (Swaminathan & Joachims, 2015a;b; Atan et al., 2018).”
Off-policy Bandits with Deficient Support (2020, methodology): “However, policy learning algorithms based on IPS and related estimators [20, 21, 24–26] require the assumption that the logging policy has full support for every policy in the policy space.”
MOReL : Model-Based Offline Reinforcement Learning (2020, background): “[12], and has applications in healthcare [33, 34, 35], recommendation systems [36, 37, 38, 39], dialogue systems [40, 19, 41], and autonomous driving [42].”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, background): “…to yield Primal AlgaeDICE is key to transforming the Lagrangian optimization over values and state-action occupancies — typical in these previous works — to an optimization over values and policies, which is more common in practice and can help generalization (e.g., Swaminathan and Joachims, 2015).”
Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, methodology): “[55] Adith Swaminathan and Thorsten Joachims.”
Learning When-to-Treat Policies (2019, methodology): “The promise of personalized data-driven decision-making has led to a surge in interest in methods that leverage observational data to help inform how and to whom interventions should be applied (Athey and Wager, 2017; Bertsimas and Kallus, 2020; Dud́ık, Erhan, Langford, and Li, 2014; Elmachtoub and ”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “Prior model-free offline RL algorithms have been designed to regularize the learned policy to be “close“ to the behavioral policy either implicitly via regularized variants of importance sampling based algorithms [47, 58, 35, 59, 41], offline actor-critic methods [53, 45, 27, 16, 64], applying uncer”
Learning Representations for Counterfactual Inference (2016, background): “Note that while in contextual bandit or robotics applications the researcher typically knows the method underlying the action choice (e.g. the policy in reinforcement learning), in observational studies we usually do not have con-trol or even a full understanding of the mechanism which chooses…”
Batch Policy Learning under Constraints (2019, methodology): “Typically, inverse propensity scoring based methods call for stochastic behavior and evaluation policies (Precup et al., 2000; Swaminathan & Joachims, 2015).”
Unbiased Learning-to-Rank with Biased Feedback (2016, methodology): “Diﬀerent from online learning via bandit algorithms and interleaving [29, 21], batch learning from ban-dit feedback (BLBF) [24] still uses randomization during feedback collection, and then performs oﬄine learning.”
A General Framework for Counterfactual Learning-to-Rank (2018, methodology): “In the case of DCG, since the rank weighting function is concave, one possible workaround is a Majorization-Minimization scheme [28] (akin to CCP): upper bound the loss function with a linear Taylor approximation at the current neural net weights, perform SGD at the level of document pairs (yi ,y) t”
BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning (2019, background): “…and functional approximator settings has long been studied (Lange et al., 2012; Strehl et al., 2010) and continues to be a highly active area of research (Swaminathan & Joachims, 2015; Jiang & Li, 2015; Thomas & Brunskill, 2016; Farajtabar et al., 2018; Irpan et al., 2019; Jaques et al., 2019).”
An Optimistic Perspective on Offline Reinforcement Learning (2019, methodology): “In our offline setup, we only assume access to samples from the behavior policy and focus on Q-learning methods without any form of importance correction, as opposed to (Swaminathan and Joachims, 2015; Liu et al., 2019).”