Doubly Robust Off-policy Value Evaluation for Reinforcement Learning

By Nan Jiang, Lihong Li. 2015

735 citations recorded by the source corpus.

We study the problem of off-policy value evaluation in reinforcement learning (RL), where one aims to estimate the value of a new policy based on data collected by a different policy. This problem is often a critical step when applying RL in real-world problems. Despite its importance, existing general methods either have uncontrolled bias or suffer high variance. In this work, we extend the doubly robust estimator for bandits to sequential decision-making problems, which gets the best of both worlds: it is guaranteed to be unbiased and can have a much lower variance than the popular importance sampling estimators. We demonstrate the estimator's accuracy in several benchmark problems, and illustrate its use as a subroutine in safe policy improvement. We also provide theoretical results on the hardness of the problem, and show that our estimator can match the lower bound in certain scenarios.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistics and Probability, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Statistics and Probability

In Reinforcement Learning

Citations in the corpus

Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback (2018, methodology): “, 2011; Swaminathan and Joachims, 2015a), or equivalently, off-policy reinforcement learning (Precup et al., 2000; Jiang and Li, 2016).”
Can Neural Machine Translation be Improved with User Feedback? (2018, methodology): “Lawrence et al. (2017b) furthermore propose the Doubly Controlled objective (DC, Equation 7) implementing the idea of doubly robust estimation (Dudı́k et al., 2011; Jiang and Li, 2016) for deterministic logs.”
Constrained Policy Optimization (2017, methodology): “When using sampling to compute policy updates, as is typically done in high-dimensional control (Duan et al., 2016), this requires off-policy evaluation, which is known to be challenging (Jiang & Li, 2015).”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “, 2016)), or have performed off-policy policy evaluation (Farajtabar et al., 2018; Jiang and Li, 2016; Precup, 2000; Thomas and Brunskill, 2016).”
Neural Approaches to Conversational AI (2018, neutral): “human conversation corpora for evaluation. Unfortunately, this problem, known as offpolicy evaluation in the RL literature, is challenging with numerous current research efforts (Precup et al., 2000; Jiang and Li, 2016; Thomas and Brunskill, 2016; Liu et al., 2018b). It is expected that off-policy t”
Machine Learning Methods That Economists Should Know About (2019, background): “A major theme of this review is that even though there are cases where using simple of-the-shelf algorithms from the ML literature can be effective (see Mullainathan and Spiess [2017] for a number of examples), there are also many cases where this is not the case.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “(6) This is known as the doubly robust estimator (Jiang and Li, 2015; Thomas and Brunskill, 2016) because it is unbiased if either πβ is known or if the model is correct.”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, background): “ue, often by constraining the actor updates to avoid excessive deviation from the data distribution [23, 41, 13, 14, 12, 21, 8, 22, 38, 26, 47]. One class of these methods utilize importance sampling [41, 47, 26, 5, 17, 12]. Another class of methods perform ofﬂine reinforcement learning via dynamic ”
Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search (2018, background): “However, a mismatch between the model and the true environment, often unavoidable in practice, can cause this approach to fail (Talvitie, 2014), resulting in policies that do not generalize to the real environment (Jiang et al., 2015).”
Evaluating Reinforcement Learning Algorithms in Observational Health Settings (2018, methodology): “…of four modern importance sampling methods on the dataset described in Section 2: per-decision importance sampling (PDIS), weighted per-decision importance sampling (WPDIS) [Precup et al., 2000], doubly-robust (DR) [Jiang and Li, 2015], and weighted doubly-robust (WDR) [Thomas and Brunskill, 2016].”
Deep-Treat: Learning Optimal Personalized Treatments From Observational Data Using Neural Networks (2018, methodology): “The doubly robust estimation techniques combine the direct and IPS methods and generate more robust counterfactual estimates (Dudı́k, Langford, and Li 2011; Jiang and Li 2016).”
Reinforcement Learning in Healthcare: A Survey (2019, background): “e there is a large volume of work in RL community that focuses on importance sampling (IS) techniques and how to trade off between bias and variance in IS-based offpolicy evaluation estimators (e.g., [317]), simply adopting these estimators in healthcare settings might be unreliable due to issues of”
Balanced Linear Contextual Bandits (2018, positive): “There is also a signiﬁcant body of research that leverages balancing for ofﬂine evaluation and learning of contextual bandit or reinforcement learning policies from logged data (Strehl et al. 2010), (Dud ´ ık, Langford, and Li 2011), (Li et al. 2012), (Dud ´ ık et al. 2014), (Li et al. 2014), (Swami”
Estimation Considerations in Contextual Bandits (2017, background): “There is also a significant body of research that leverages balancing for offline evaluation and learning of contextual bandit or reinforcement learning policies from logged data [57, 28, 44, 27, 43, 59, 37, 60, 10, 38, 64, 26, 39, 58, 66].”
Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning (2016, methodology): “We also compare to the guided importance sampling doubly robust (DR) estimator (Jiang & Li, 2015).”
Estimating Counterfactual Treatment Outcomes over Time Through Adversarially Balanced Representations (2020, methodology): “These methods use algorithms based on importance sampling (Precup, 2000; Thomas et al., 2015; Guo et al., 2017), action-value function approximation (model based) (Hallak et al., 2015) or doubly robust combination of both approaches (Jiang & Li, 2015).”