Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning

By Cameron Voloshin, Hoang Minh Le, Nan Jiang, Yisong Yue. 2019

180 citations recorded by the source corpus.

Off-policy policy evaluation (OPE) is the problem of estimating the online performance of a policy using only pre-collected historical data generated by another policy. Given the increasing interest in deploying learning-based methods for safety-critical applications, many recent OPE methods have recently been proposed. Due to disparate experimental conditions from recent literature, the relative performance of current OPE methods is not well understood. In this work, we present the first comprehensive empirical analysis of a broad suite of OPE methods. Based on thousands of experiments and detailed empirical analyses, we offer a summarized set of guidelines for effectively using OPE in practice, and suggest directions for future research.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Software, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Software

In Reinforcement Learning

In Evaluation & Benchmarks

Citations in the corpus

Offline RL Without Off-Policy Evaluation (2021, background): “See Voloshin et al. [2019], Wang et al. [2021] for more comprehensive examinations of policy evaluation and some evidence that this simple ﬁtted Q iteration approach is reasonable.”
Benchmarks for Deep Off-Policy Evaluation (2021, methodology): “Finally, all evaluations in Voloshin et al. (2019) are performed using the MSE metric, and they do not provide standardized datasets.”
Hyperparameter Selection for Offline Reinforcement Learning (2020, methodology): “Pseudocode for it can be found in Algorithm 1, and a simpliﬁed version of the code we use can be found in Appendix D. FQE was shown to work well in a recent suite of experiments on relatively simple problems (Voloshin et al., 2019).”
Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error (2022, methodology): “The success of FQE for policy evaluation is supported by many examples in the literature (Voloshin et al., 2021; Fu et al., 2021; Fujimoto et al., 2021), as well as control (Mnih et al.”
Minimax Confidence Interval for Off-Policy Evaluation and Policy Optimization (2020, methodology): “[31] Cameron Voloshin, Hoang M Le, Nan Jiang, and Yisong Yue.”
Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization (2021, methodology): “…that we focus on in this paper lies within the class of algorithms referred to as the direct method (Kostrikov and Nachum, 2020; Dudík et al., 2011; Voloshin et al., 2019), which approximate the value of a new policy by either explicitly or implicitly estimating the transition and reward functions…”
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems (2022, background): “[63] present a comprehensive empirical study of several different methods.”
NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning (2021, methodology): “Offline policy evaluation (OPE) extends off-policy policy evaluation (OPPE) to offline setting [13, 14] and only uses the dataset to evaluate a policy, thus it can be applied as a tool for offline policy validation.”
Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation (2020, methodology): “We present the computed intervals of FQE and MB in Figure 3.”