Benchmarking Batch Deep Reinforcement Learning Algorithms

By Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, Joelle Pineau. 2019

218 citations recorded by the source corpus.

Widely-used deep reinforcement learning algorithms have been shown to fail in the batch setting--learning from a fixed data set without interaction with the environment. Following this result, there have been several papers showing reasonable performances under a variety of environments and batch settings. In this paper, we benchmark the performance of recent off-policy and batch reinforcement learning algorithms under unified settings on the Atari domain, with data generated by a single partially-trained behavioral policy. We find that under these conditions, many of these algorithms underperform DQN trained online with the same amount of data, as well as the partially-trained behavioral policy. To introduce a strong baseline, we adapt the Batch-Constrained Q-learning algorithm to a discrete-action setting, and show it outperforms all existing algorithms at this task.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Electrical and Electronic Engineering, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Electrical and Electronic Engineering

In Reinforcement Learning

In Evaluation & Benchmarks

Citations in the corpus

Offline RL Without Off-Policy Evaluation (2021, methodology): “Some authors have proposed explicit constraints on the learned policy to only select actions where ( s, a ) has sufﬁcient support under the data generating distribution [Fujimoto et al., 2018a, 2019, Laroche et al., 2019].”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “Some methods have modified the set of valid actions based on counts [Laroche et al., 2019] or the learned behavior policy [Fujimoto et al., 2019a].”
Regularized Behavior Value Estimation (2021, methodology): “…broad categories: 1) policy-constraint approaches regularize the learned policy to stay close to the behavior policy either explicitly or implicitly (Fujimoto et al., 2019b; Kumar et al., 2019; Jaques et al., 2019; Siegel et al., 2020; Wang et al., 2020; Ghasemipour et al., 2020), 2) value-based…”
Critic Regularized Regression (2020, background): “Overall, our results confirm previously published results that the naive application of off-policy RL algorithms can fail in the offline RL settings [9, 30]: D4PG performs well on some of the DCS domains, but fails on the domains with higher dimensional action spaces (humanoid-run, and problems on t”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, background): “It is a popular benchmark to measure the progress of online RL methods, and Atari has recently also become a standard benchmark for offline RL methods (Agarwal et al., 2020; Fujimoto et al., 2019) as well.”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “Atari has been an established benchmark for offline-RL (Agarwal et al., 2019; Fujimoto et al., 2019).”
QPLEX: Duplex Dueling Multi-Agent Q-Learning (2020, background): “This ability not only provides QPLEX with high stability and sample efficiency but also with opportunities to efficiently utilize multi-source offline data without additional online exploration (Fujimoto et al., 2019; Fu et al., 2020; Levine et al., 2020; Yu et al., 2020).”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, methodology): “Table 5 and Table 6 shows the tunable hyper-parameters of BCQ and BRAC, respectively.”
Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error (2022, methodology): “Previous work has observed that the value function of FQE methods can diverge when computed offline due to poor estimates in the target (Fujimoto et al., 2019b;a).”
Overcoming Model Bias for Robust Offline Deep Reinforcement Learning (2020, methodology): “…methods have been developed for continuous state and action spaces, even more algorithms have been developed for the ofﬂine setting with discrete actions (Laroche, Trichelair, and Combes 2017; Dabney et al. 2018; Agarwal, Schuurmans, and Norouzi 2019; Jaques et al. 2019; Fujimoto et al. 2019).”
NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning (2021, methodology): “ut the out-of-data distribution, and constrain the RL agent towards the behavior policy that produces the ofﬂine data, or use ensemble models to measure the uncertainty of out-of-data demonstrations [Fujimoto et al., 2019b; Yu et al., 2020]. Although these ofﬂine RL approaches have achieved great su”
RL Unplugged: A Suite of Benchmarks for Ofﬂine Reinforcement Learning (2020, methodology): “This highlights another key problem in online and ofﬂine RL. Papers do not include key baselines because the authors were not able to reproduce them, see eg [Peng et al., 2019, Fujimoto et al., 2019].”
BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning (2019, methodology): “Batch algorithms that apply to discrete actions space have been benchmarked on the Atari environments, showing that a discrete variant of BCQ has robust performance and outperform a number of other methods [7].”
Batch Reinforcement Learning Through Continuation Method (2021, methodology): “The dataset D is used to train each ofﬂine RL agent. and because We compare with BCQ [13], REM [2] CQL [22] they are recently proposed ofﬂine RL algorithms and work well on Atari domain.”
Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning (2021, background): “To solve these issues, offline RL is an excellent choice to deal with practical problems [3, 24, 35, 42, 15, 28, 4, 23, 54, 12], aiming at learning from a fixed dataset without interaction with environments.”
An Optimistic Perspective on Offline Reinforcement Learning (2019, methodology): “Motivated by the so-called final buffer setting in Fujimoto et al. (2019b) (Appendix A.2), we train a DDPG agent on continuous control MuJoCo tasks (Todorov et al., 2012) for 1 million time steps and store all of the experienced transitions.”
Towards Understanding Linear Value Decomposition in Cooperative Multi-Agent Q-Learning (2020, unknown): “Differing with other related work studying extrapolation error [38–40], we aim to adopt a diverse dataset simulating a uniform data distribution, which makes the function expressiveness of value decomposition become the dominant factor to be investigated.”
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems (2022, background): “In practice, this ensemble offers very little diversity, causing the uncertainty to be underestimated and making us more prone to take OOD actions [58].”