Striving for Simplicity in Off-policy Deep Reinforcement Learning

By Rishabh Agarwal, Dale Schuurmans, Mohammad Norouzi. 2019

71 citations recorded by the source corpus.

Reflecting on the advances of off-policy deep reinforcement learning (RL) algorithms since the development of DQN in 2013, it is important to ask: are the complexities of recent off-policy methods really necessary? In an attempt to isolate the contributions of various factors of variation in off-policy deep RL and to help design simpler algorithms, this paper investigates a set of related questions: First, can effective policies be learned given only access to logged offline experience? Second, how much of the benefits of recent distributional RL algorithms is attributed to improvements in exploration versus exploitation behavior? Third, can simpler off-policy RL algorithms outperform distributional RL without learning explicit distributions over returns? This paper uses a batch RL experimental setup on Atari 2600 games to investigate these questions. Unexpectedly, we find that batch RL algorithms trained solely on logged experiences of a DQN agent are able to significantly outperform online DQN. Our experiments suggest that the benefits of distributional RL mainly stem from better exploitation. We present a simple and novel variant of ensemble Q-learning called Random Ensemble Mixture (REM), which enforces optimal Bellman consistency on random convex combinations of the Q-heads of a multi-head Q-network. The batch REM agent trained offline on DQN data outperforms the batch QR-DQN and online C51 algorithms.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “Others have tackled this problem using deep learning, but have not used KL-control (Liu et al., 2019; Gelada and Bellemare, 2019; Bhatt et al., 2019; Kumar et al., 2019; Agarwal et al., 2019; Fujimoto et al., 2018; Ghasemipour et al., 2020).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Some prior works on offline RL have also noted that, for some types of datasets, conventional dynamic programming algorithms, such as deep Q-learning or deterministic actor-critic algorithms, can in fact work reasonably well (Agarwal et al., 2019).”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “…RL methods are designed with two main ingredients: (1) constraining the learned policy to be closer to the behavioral policy either explicitly [18, 35, 70, 29, 48] or implicitly [51, 57], and (2) applying uncertainty quantiﬁcation techniques, such as ensembles, to stabilize Q-functions [1, 35, 70].”
MOReL : Model-Based Offline Reinforcement Learning (2020, methodology): “Recent studies show that directly using off-policy RL algorithms with an offline dataset yields poor results due to distribution shift and function approximation errors [15, 16, 17].”
D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, methodology): “…those from deterministic expert policies, are particularly problematic for offline RL algorithms and may cause divergence both empirically [Fujimoto et al., 2018, Kumar et al., 2019] and theoretically [Munos, 2003, Farahmand et al., 2010, Kumar et al., 2019, Agarwal et al., 2019a, Du et al., 2020].”
Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning (2020, background): “…function approximators has mainly considered the “online” or “growing batch” settings, where additional exploration data is collected (Ernst et al., 2005; Riedmiller, 2005; Ormoneit & Sen, 2002); though some success for batch RL in discrete domains has been reported (Agarwal et al., 2019).”
A Framework for Data-Driven Robotics (2019, background): “Confirming previous findings [65], [67], we found distributional value functions to be very helpful.”
Behavior Regularized Offline Reinforcement Learning (2019, methodology): “value function (Q-value function) learned with function approximators, as suggested by Sutton (1995); Baird (1995); Tsitsiklis & Van Roy (1997); Van Hasselt et al. (2018). To remedy this issue, two types of approaches have been proposed recently: 1) Agarwal et al.”
Regularized Behavior Value Estimation (2021, background): “Regularized Behavior Value Estimation is designed to avoid over-estimation from extrapolation during training and deployment. functions; e.g. (Agarwal et al., 2019b).”
Critic Regularized Regression (2020, background): “This is also consistent with the results of [2] who find that standard off-policy RL algorithms can perform well in the offline setting in some cases; presumably when the state-action space of the domain is well covered in the dataset.”
Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics (2020, methodology): “Embodiments of this approach are based on dropout Anschel et al. (2017), employing previous Q-function approximations (Ghavamzadeh et al.”
IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data (2019, background): “arbitrary off-policy data for policy learning without collecting additional experience [3, 11, 16, 21, 23].”
Benchmarking Batch Deep Reinforcement Learning Algorithms (2019, methodology): “Contrary to these results, Agarwal et al. [2019] showed that using the entire history of a deep reinforcement learning agent as a batch (50 million time steps), standard deep reinforcement learning algorithms could reach a comparable performance to an online algorithm.”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “More recently, a related but distinct problem known as ofﬂine RL has gained popularity (Levine et al., 2020; Wu et al., 2019; Agarwal et al., 2019; Kumar et al., 2020).”
Overcoming Model Bias for Robust Offline Deep Reinforcement Learning (2020, methodology): “…methods have been developed for continuous state and action spaces, even more algorithms have been developed for the ofﬂine setting with discrete actions (Laroche, Trichelair, and Combes 2017; Dabney et al. 2018; Agarwal, Schuurmans, and Norouzi 2019; Jaques et al. 2019; Fujimoto et al. 2019).”
DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learning (2020, background): “, [23] is a thorough study of Q-learning algorithms, [4] identifies the key concept of overestimation bias in Q-learning, [24] shows that offline Q-learning, i.”
Scaling data-driven robotics with reward sketching and batch reinforcement learning (2019, methodology): “Our choice of distributional RL is partly motivated by the success of this method for batch RL in Atari [1].”