Benchmarks for Deep Off-Policy Evaluation

By Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyun Wang, Alexander Novikov, Mengjiao Yang, Michael R. Zhang, Yutian Chen, Aviral Kumar and 3 more. 2021

117 citations recorded by the source corpus.

Off-policy evaluation (OPE) holds the promise of being able to leverage large, offline datasets for both obtaining and selecting complex policies for decision making. The ability to perform evaluation offline is particularly important in many real-world domains, such as healthcare, recommender systems, or robotics, where online data collection is an expensive and potentially dangerous process. Being able to accurately evaluate and select high-performing policies without requiring online interaction could yield significant benefits in safety, time, and cost for these applications. While many OPE methods have been proposed in recent years, comparing results between works is difficult because there is currently a lack of a comprehensive and unified benchmark. Moreover, it is difficult to measure how far algorithms have progressed, due to the lack of challenging evaluation tasks. In order to address this gap, we propose a new benchmark for off-policy evaluation which includes tasks on a range of challenging, high-dimensional control problems, with wide selections of datasets and policies for performing policy selection. The goal of of our benchmark is to provide a standardized measure of progress that is motivated from a set of principles designed to challenge and test the limits of existing OPE methods. We perform a comprehensive evaluation of state-of-the-art algorithms, and we will provide open-source access to all data and code to foster future research in this area.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

In Evaluation & Benchmarks

Citations in the corpus

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, background): “We hope that our datasets can help supplement other efforts [29].”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “However, since neither paper make mention of a training/validation split, we can only assume these changes were made with their evaluation datasets in mind (D4RL, in this instance), and remark there is insufficient evidence that these changes may be universal.”
Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning (2019, methodology): “While many recent methods are built on sound mathematical principles, a notable gap in the current literature is a standard for benchmarking empirical studies, with perhaps a notable exception from the recent DOPE [18] and D4RL benchmarks [17].”
Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization (2021, methodology): “These metrics capture different desirable properties of OPE methods (Fu et al., 2021); more details about how they are computed are in the appendix.”
NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning (2021, methodology): “We also plot the estimate vs ground-truth return as [14, 6] in the Appendix G.”
RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning (2021, background): “Ofﬂine Policy Estimation research [16] will likely beneﬁt from such standardization of datasets.”
Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error (2022, methodology): “The success of FQE for policy evaluation is supported by many examples in the literature (Voloshin et al., 2021; Fu et al., 2021; Fujimoto et al., 2021), as well as control (Mnih et al.”
RvS: What is Essential for Offline RL via Supervised Learning? (2021, background): “Fully automated tuning of hyperparameters remains an open question [13, 33].”
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems (2022, methodology): “[67] opt to use the absolute error instead of the MSE to increase robustness to outliers.”