D4RL: Datasets for Deep Data-Driven Reinforcement Learning

By Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, Sergey Levine. 2020

1,854 citations recorded by the source corpus.

The offline reinforcement learning (RL) problem, also known as batch RL, refers to the setting where a policy must be learned from a static dataset, without additional online data collection. This setting is compelling as potentially it allows RL methods to take advantage of large, pre-collected datasets, much like how the rise of large datasets has fueled results in supervised learning in recent years. However, existing online RL benchmarks are not tailored towards the offline setting, making progress in offline RL difficult to measure. In this work, we introduce benchmarks specifically designed for the offline setting, guided by key properties of datasets relevant to real-world applications of offline RL. Examples of such properties include: datasets generated via hand-designed controllers and human demonstrators, multi-objective datasets where an agent can perform different tasks in the same environment, and datasets consisting of a mixtures of policies. To facilitate research, we release our benchmark tasks and datasets with a comprehensive evaluation of existing algorithms and an evaluation protocol together with an open-source codebase. We hope that our benchmark will focus research effort on methods that drive improvements not just on simulated tasks, but ultimately on the kinds of real-world problems where offline RL will have the largest impact.

Subfields: Artificial Intelligence, Automotive Engineering, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Reinforcement Learning

In Evaluation & Benchmarks

Citations in the corpus

Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, methodology): “In this section, we consider the continuous control tasks from the D4RL benchmark (Fu et al., 2020).”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “2021) on the AntMaze navigation tasks (Fu et al., 2020), for which there is only a sparse reward upon reaching the goal state.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “However, even this formulation does not fully evaluate capabilities of offline reinforcement learning methods, and the statistics of the training data have a considerable effect on the difficult of offline RL (Fu et al., 2020), including how concentrated the data distribution is around a specific se”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “We first evaluate actor-critic CQL, using CQL(H) from Algorithm 1, on continuous control datasets from the D4RL benchmark [12].”
Offline Reinforcement Learning with Implicit Q-Learning (2021, methodology): “50 E p i s d e e t u In the former category, we compare to CQL (Kumar et al., 2020), TD3+BC (Fu-jimoto & Gu, 2021), and AWAC (Nair et al., 2020). n We compare to methods that are representative of both multi-step dynamic programming and one-step approaches.”
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, methodology): “Benchmark datasets [18, 19] have also been proposed recently for batch (ofﬂine) reinforcement learning.”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “Our results suggest that MOPO substantially outperforms these prior methods on the ofﬂine RL benchmark D4RL [17] as well as on ofﬂine RL problems where the agent must generalize to out-of-distribution states in order to succeed.”
Implicit Behavioral Cloning (2021, background): “The following results and discussions are101 organized by task domain – each evaluating a unique set of desired properties for policy learning (Table 1).102 All tasks are characterized by discontinuities and require generalization (e.g., extrapolation) to some degree.103 D4RL [17] is a recent benchm”
Offline RL Without Off-Policy Evaluation (2021, methodology): “Following Fu et al. [2020] and others in this line of work, we allow access to the environment to tune a small (< 10) set of hyperparameters.”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “, 2020] as well as the current trend of reporting only the mean-value of the final policy in offline benchmarking [Fu et al., 2020].”
Regularized Behavior Value Estimation (2021, background): “In contrast, in the offline setting (Fu et al., 2020; Fujimoto 1DeepMind, London, UK 2Google Brain, Toronto, Canada.”
Offline Reinforcement Learning with Fisher Divergence Critic Regularization (2021, methodology): “For example, a choice of Based on a similar idea, CQL (Kumar et al., 2020) extends the standard critic loss J ( Q θ ) in eq.”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, methodology): “The resultant method, Continuous Doubly Constrained Batch RL (CDC) exhibits state-of-the-art performance across 32 continuous control tasks from the D4RL benchmark [14] demonstrating the usefulness of our regularizers for batch RL.”
EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, methodology): “We note that with our proposed autoregressive models, the re-sults for BCQ and BEAR are equal to or improved upon the values reported in the D4RL benchmark (Fu et al., 2020b) (green horizontal lines).”
Provable Representation Learning for Imitation with Contrastive Fourier Features (2021, background): “The limited availability of target demonstrations stands in contrast to the recent proliferation of large offline datasets for sequential decision making [28, 19, 7, 21].”
Hyperparameter Selection for Imitation Learning (2021, methodology): “For the Adroit environments, we use the “expert” datasets from the D4RL dataset (Fu et al., 2020).”
Benchmarks for Deep Off-Policy Evaluation (2021, methodology): “Maze2D and AntMaze are two maze navigation tasks originally proposed in D4RL (Fu et al., 2020).”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “On the D4RL experiments, we found that β = 0 .”
Representation Matters: Offline Pretraining for Sequential Decision Making (2021, methodology): “For offline datasets, we leverage the Gym-MuJoCo datasets from D4RL (Fu et al., 2020), which provide a diverse set of datasets from continuous control simulated robotic environments.”