Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning

By Philip S. Thomas, Emma Brunskill. 2016

658 citations recorded by the source corpus.

In this paper we present a new way of predicting the performance of a reinforcement learning policy given historical data that may have been generated by a different policy. The ability to evaluate a policy from historical data is important for applications where the deployment of a bad policy can be dangerous or costly. We show empirically that our algorithm produces estimates that often have orders of magnitude lower mean squared error than existing methods---it makes more efficient use of the available data. Our new estimator is based on two advances: an extension of the doubly robust estimator (Jiang and Li, 2015), and a new way to mix between model based estimates and importance sampling based estimates.

Subfields: Artificial Intelligence, Management Science and Operations Research, Software, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Software

In Reinforcement Learning

In Evaluation & Benchmarks

Citations in the corpus

ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, methodology): “…of the policy parameters [36, 37], by enforcing the new policy to stay Importance Sampling for Reinforcement Learning In RL, IS is generally used for sample ef-ﬁciency purposes: in off-policy policy gradient methods, IS allows to re-use previously sampled sequences more than once [48, 44, 12].”
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback (2018, methodology): “This degenerate behavior can be avoided by reweighting using a multiplicative control variate (Kong, 1992; Precup et al., 2000; Jiang and Li, 2016; Thomas and Brunskill, 2016).”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “…to explore and collect more data (Degris et al., 2012; Ried-miller, 2005) such as learning online from an out-dated replay buffer (e.g. (Munos et al., 2016)), or have performed off-policy policy evaluation (Fara-jtabar et al., 2018; Jiang and Li, 2016; Precup, 2000; Thomas and Brunskill, 2016).”
Deep Reinforcement Learning that Matters (2017, background): “Several works have begun investigating policy evaluation methods for the purposes of safe RL (Thomas and Brunskill 2016; Thomas, Theocharous, and Ghavamzadeh 2015), but further work is needed in signiﬁcance testing and statistical analysis.”
Neural Approaches to Conversational AI (2018, neutral): “Unfortunately, this problem, known as off-policy evaluation in the RL literature, is challenging with numerous current research efforts (Precup et al., 2000; Jiang and Li, 2016; Thomas and Brunskill, 2016; Liu et al., 2018c).”
Machine Learning Methods That Economists Should Know About (2019, background): “Generalized random forests can be thought of as a generalization of local maximum likelihood, introduced by Tibshirani and Hastie [1987], but where kernel weighting functions are used to weight nearby observations more heavily than observations distant from a particular test point. A weakness of for”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “(6) This is known as the doubly robust estimator (Jiang and Li, 2015; Thomas and Brunskill, 2016) because it is unbiased if either πβ is known or if the model is correct.”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, neutral): “One class of these methods utilize importance sampling [43, 51, 28, 6, 19, 14].”
Evaluating Reinforcement Learning Algorithms in Observational Health Settings (2018, methodology): “, 2000], doubly-robust (DR) [Jiang and Li, 2015], and weighted doubly-robust (WDR) [Thomas and Brunskill, 2016].”
Reinforcement Learning in Healthcare: A Survey (2019, background): “While there is a large volume of work in RL that focuses on importance sampling (IS) techniques [221] and how to trade off between bias and variance in IS-based off-policy evaluation estimators, directly adopting these estimators in healthcare settings might be unreliable due to issues of sparse rew”
Balanced Linear Contextual Bandits (2018, neutral): “2014), (Swaminathan and Joachims 2015), (Jiang and Li 2016), (Thomas and Brunskill 2016), (Athey and Wager 2017), (Kallus 2017), (Wang, Agarwal, and Dudı́k 2017), (Deshpande et al.”
Estimation Considerations in Contextual Bandits (2017, background): “There is also a significant body of research that leverages balancing for offline evaluation and learning of contextual bandit or reinforcement learning policies from logged data [57, 28, 44, 27, 43, 59, 37, 60, 10, 38, 64, 26, 39, 58, 66].”
Off-policy Bandits with Deficient Support (2020, methodology): “With terabytes of log data readily available in many online systems, a range of algorithms has been proposed for batch learning from such logged contextual-bandit feedback [5, 6, 20–22, 24, 26].”
Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning (2020, methodology): “We will first use a simple MDP from [36] to highlight the IPS drawback we previously mentioned in Section 1.”