RL Unplugged: Benchmarks for Offline Reinforcement Learning

By Caglar Gulcehre, Ziyun Wang, Alexander Novikov, T. Paine, Sergio Gomez Colmenarejo, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Jaymin Mankowitz, Cosmin Paduraru and 8 more. 2020

74 citations recorded by the source corpus.

Offline methods for reinforcement learning have a potential to help bridge the gap between reinforcement learning research and real-world applications. They make it possible to learn policies from offline datasets, thus overcoming concerns associated with online data collection in the real-world, including cost, safety, or ethical concerns. In this paper, we propose a benchmark called RL Unplugged to evaluate and compare offline RL methods. RL Unplugged includes data from a diverse range of domains including games (e.g., Atari benchmark) and simulated motor control problems (e.g., DM Control Suite). The datasets include domains that are partially or fully observable, use continuous or discrete actions, and have stochastic vs. deterministic dynamics. We propose detailed evaluation protocols for each domain in RL Unplugged and provide an extensive analysis of supervised learning and offline RL methods using these protocols. We will release data for all our tasks and open-source all algorithms presented in this paper. We hope that our suite of benchmarks will increase the reproducibility of experiments and make it possible to study challenging tasks with a limited computational budget, thus making RL research both more systematic and more accessible across the community. Moving forward, we view RL Unplugged as a living benchmark suite that will evolve and grow with datasets contributed by the research community and ourselves. Our project page is available on this https URL.

Subfields: Artificial Intelligence, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Evaluation & Benchmarks

Citations in the corpus

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, background): “This makes it difficult to select the best trained model [19, 28, 29].”
Implicit Behavioral Cloning (2021, neutral): “On offline RL benchmarks, prior works’ implementations of BC already show reasonably competitive results with offline RL algorithms [17, 58].”
Offline RL Without Off-Policy Evaluation (2021, methodology): “Gulcehre et al. [2021] studies the discrete action setting and ﬁnds that a one-step algorithm (which they call “behavior value estimation”) outperforms prior work on Atari games and other discrete action tasks from the RL Unplugged benchmark [Gulcehre et al., 2020].”
Regularized Behavior Value Estimation (2021, methodology): “For the Atari domain we used the RL Unplugged Atari benchmark (Gulcehre et al., 2020).”
Provable Representation Learning for Imitation with Contrastive Fourier Features (2021, background): “The limited availability of target demonstrations stands in contrast to the recent proliferation of large offline datasets for sequential decision making [28, 19, 7, 21].”
Hyperparameter Selection for Imitation Learning (2021, background): “Some recent benchmarks (Gulcehre et al., 2020; Fu et al., 2020) also propose evaluation protocols where HPs are selected by tuning on only a subset of environments.”
Benchmarks for Deep Off-Policy Evaluation (2021, methodology): “These two domains are constructed using two benchmarks previously proposed for offline reinforcement learning: RL Unplugged (Gulcehre et al., 2020) and D4RL (Fu et al., 2020), and reflect the challenges found within them.”
Accelerating Reinforcement Learning with Learned Skill Priors (2020, methodology): “This data can be collected using previously trained agents across a diverse set of tasks [6, 8], through agents autonomously exploring their environment [12, 30],”
Hyperparameter Selection for Offline Reinforcement Learning (2020, methodology): “included in Gulcehre et al. (2020). For all tasks in this domain we use a feature representation of the MDP state, including proprioceptive information such as joint positions and velocities.”
TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations (2021, neutral): “More recently, ofﬂine reinforcement learning has attracted attention because it intends to learn a better policy than the behavior policies which generate the dataset (Fu et al., 2020; Gulcehre et al., 2020; Qin et al., 2021).”
Offline Reinforcement Learning from Images with Latent Space Models (2020, background): “…(Fujimoto et al., 2018; Agarwal et al., 2019; Kumar et al., 2020) or propose benchmarks that include simulated locomotion tasks with pixel inputs (Gulcehre et al., 2020); our focus in this work is on continuous robot control from high-dimensional perceptual inputs in both simulation and the real…”
Behavior Transformers: Cloning k modes with one stone (2022, unknown): “As behavior cloning algorithms are generally solving a fully supervised learning problem, they tend to be faster and simpler than reinforcement learning or ofﬂine RL algorithms and in some cases show competitive results [26, 33].”
Dataset Distillation by Matching Training Trajectories (2022, methodology): “Such methods (behavior cloning in particular) have been shown to work well in ofﬂine settings [9, 12].”
Skill-based Meta-Reinforcement Learning (2022, background): “…at scale, e . g . through autonomous exploration (Hausman et al., 2018; Sharma et al., 2020; Dasari et al., 2019), human teleoperation (Schaal et al., 2005; Gupta et al., 2019; Mandlekar et al., 2018; Lynch et al., 2020), or from previously trained agents (Fu et al., 2020; Gulcehre et al., 2020).”
Model-Based Offline Planning (2020, methodology): “We begin by a complete description of the MBOP algorithm in Section 2, we then look at how similar offline works compare to MBOP in Section 3, and then demonstrate MBOP performance in various benchmarked offline control tasks from RL Unplugged [3] and D4RL [4].”
RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning (2021, methodology): “From human [31] to synthetic [19] demonstrations.”
Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization (2021, methodology): “To test dynamic models for policy optimization, we implement the two methods discussed in Section 5.4 on top of CRR exp , one of the CRR variants (Wang et al., 2020).”