Learning from Logged Implicit Exploration Data

By Alexander L. Strehl, J. Langford, Lihong Li, Sham M. Kakade. 2010

267 citations recorded by the source corpus.

We provide a sound and consistent foundation for the use of nonrandom exploration data in "contextual bandit" or "partially labeled" settings where only the value of a chosen action is learned. The primary challenge in a variety of settings is that the exploration policy, in which "offline" data is logged, is not explicitly known. Prior solutions here require either control of the actions during the learning process, recorded random exploration, or actions chosen obliviously in a repeated manner. The techniques reported here lift these restrictions, allowing the learning of a policy for choosing actions given features from historical data where no randomization occurred or was logged. We empirically verify our solution on two reasonably sized sets of real-world data obtained from Yahoo!.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Can Neural Machine Translation be Improved with User Feedback? (2018, background): “To our surprise, the bandit-to-supervised conversion proved to be very hard to beat, despite theoretical indications of poor generalization for exploration-free learning from logged data (Langford et al., 2008; Strehl et al., 2010).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Policy learning from logged, offline data has included studies on optimizing newspaper article click-through-rates (Strehl et al., 2010; Garcin et al., 2014), advertisement ranking on search pages (Bottou et al.”
Recommendations as Treatments: Debiasing Learning and Evaluation (2016, background): “Furthermore, note that there are several promising directions for further improving performance, like propensity clipping (Strehl et al., 2010), doubly-robust estimation (Dud ´ ık et al., 2011), and the use of improved methods for propensity estimation (McCaffrey et al., 2004).”
Counterfactual reasoning and learning systems: the example of computational advertising (2012, background): “ of 1/n. This works poorly when parts of the state space are left unexplored. A series of papers on the evaluation of contextual bandits algorithms appeared while our ad placement work was under way. Strehl et al. (2010) describe a capped estimator for contextual bandits, unfortunately in a setup th”
Balanced Linear Contextual Bandits (2018, background): “…body of research that leverages balancing for ofﬂine evaluation and learning of contextual bandit or reinforcement learning policies from logged data (Strehl et al. 2010), (Dud ´ ık, Langford, and Li 2011), (Li et al. 2012), (Dud ´ ık et al. 2014), (Li et al. 2014), (Swaminathan and Joachims…”
Estimation Considerations in Contextual Bandits (2017, background): “There is also a significant body of research that leverages balancing for offline evaluation and learning of contextual bandit or reinforcement learning policies from logged data [57, 28, 44, 27, 43, 59, 37, 60, 10, 38, 64, 26, 39, 58, 66].”
Balanced Policy Evaluation and Learning (2017, background): “Unlike [1, 7, 13, 41, 45], our (nonconvex) policy optimization problem does not reduce to weighted classification precisely because our weights are not multiplies of ⇡Ti(Xi) (but therefore our weights also lead to better performance).”
Counterfactual Estimation and Optimization of Click Metrics for Search Engines (2014, neutral): “If such a condition cannot be met (say, due to system constraints), one can still use max { p min , p a } to replace p a in Equation 1, as done by other authors [29].”
Doubly Robust Policy Evaluation and Learning (2011, methodology): “ context refers to additional information about the user or patient. Here, we focus on the ofﬂine version: we assume access to historic data, but no ability to gather new data (Langford et al., 2008; Strehl et al., 2011). Two kinds of approaches address ofﬂine learning in contextual bandits. The ﬁrs”
MOReL : Model-Based Offline Reinforcement Learning (2020, background): “[12], and has applications in healthcare [33, 34, 35], recommendation systems [36, 37, 38, 39], dialogue systems [40, 19, 41], and autonomous driving [42].”
GenDICE: Generalized Offline Estimation of Stationary Values (2020, background): “…sampling (IS) has has been explored in the contextual bandits (Strehl et al., 2010; Dudı́k et al., 2011; Wang et al., 2017), and episodic RL settings (Murphy et al., 2001; Precup et al., 2001), achieving many empirical successes (e.g., Strehl et al., 2010; Dudı́k et al., 2011; Bottou et al., 2013).”
Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation (2019, methodology): “The second approach is to use importance sampling to correct the sampling bias in off-policy data, so that an (almost) unbiased estimator can be obtained (Liu, 2001; Strehl et al., 2010; Bottou et al., 2013).”
Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, background): “More importantly, on-policy methods are not applicable to many practical RL problems [52, 8, 55] for which ofﬂine logged data already exists before policy learning proceeds.”
Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation (2018, background): “IS-based off-policy value estimation has seen a lot of interest recently for short-horizon problems, including contextual bandits [26, 13, 7, 42], and achieved many empirical successes [7, 34].”
Learning Representations for Counterfactual Inference (2016, background): “…diverse areas, for example off-policy evaluation in reinforcement learning (Sutton & Barto, 1998), learning from “logged implicit exploration data” (Strehl et al., 2010) or “logged bandit feedback” (Swaminathan & Joachims, 2015), and in understanding and designing complex real world ad-placement…”