Rainbow: Combining Improvements in Deep Reinforcement Learning

By Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, David Silver. 2017

2,665 citations recorded by the source corpus.

The deep reinforcement learning community has made several independent improvements to the DQN algorithm. However, it is unclear which of these extensions are complementary and can be fruitfully combined. This paper examines six extensions to the DQN algorithm and empirically studies their combination. Our experiments show that the combination provides state-of-the-art performance on the Atari 2600 benchmark, both in terms of data efficiency and final performance. We also provide results from a detailed ablation study that shows the contribution of each component to overall performance.

Subfields: Artificial Intelligence, Mechanical Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Mechanical Engineering

In Reinforcement Learning

Citations in the corpus

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “The idea of using a slow-moving average target network to produce stable targets for the online network was inspired by deep RL [50, 51, 52, 53].”
On Empirical Comparisons of Optimizers for Deep Learning (2019, neutral): “In reinforcement learning, Hessel et al. (2017) set (cid:15) = 1 .”
Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, background): “But these methods can be notoriously unstable, and in practice, require a variety of stabilization techniques to ensure more consistent performance (Hasselt et al., 2016; Wang et al., 2016; Munos et al., 2016; Hessel et al., 2017; Fujimoto et al., 2018; Nachum et al., 2018; Fu et al., 2019).”
Mastering Atari, Go, chess and shogi by planning with a learned model (2019, methodology): “[17] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver.”
Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “They are used in many reinforcement learning agents (Mnih et al., 2016; Schulman et al., 2017; Hessel et al., 2017).”
When to use parametric models in reinforcement learning? (2019, methodology): “We used the fairly standard convolutional Q network from Hessel et al. (2018b). We have not tried to exhaustively tune the algorithm and we do not doubt that the algorithm can be made even more data efficient by fine-tuning its hyper-parameters.”
Model-Based Reinforcement Learning for Atari (2019, methodology): “The combination of reinforcement learning and deep models then enabled RL algorithms to learn to play Atari games directly from images of the game screen, using variants of the DQN algorithm (Mnih et al., 2013; 2015; Hessel et al., 2017) and actor-critic algorithms (Mnih et al.”
Observe and Look Further: Achieving Consistent Performance on Atari (2018, methodology): “While we follow many of the practices commonly applied when training on the ALE, our experimental setup differs in a few key aspects from the defaults [13, 6].”
Distributed Prioritized Experience Replay (2018, methodology): “As a result, prioritized experience replay is used in many agents, such as Prioritized Dueling DQN (Wang et al., 2016), UNREAL (Jaderberg et al., 2017), DQfD (Hester et al., 2017), and Rainbow (Hessel et al., 2017).”
Learning and Querying Fast Generative Models for Reinforcement Learning (2018, background): “In spite of signiﬁcant progress (Hessel et al., 2017), some games are still considered very challenging environments for RL agents, e.g. MS PACMAN, especially when not using any privileged information.”
Wireless Network Intelligence at the Edge (2018, background): “Such distributional RL outperforms state-of-the-art RL methods including deep Q-learning and asynchronous actor-critic RL [144], [145].”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “Multi-step temporal difference (TD) learning [127, 128] improved both DQNs and policy gradients [47, 80].”
Meta-Gradient Reinforcement Learning (2018, methodology): “Finally, we compare against the state-of-the-art agent trained on Atari games, namely Rainbow [Hessel et al., 2018], which combines DQN [Mnih et al., 2015] with double Q-learning [van Hasselt et al., 2016, van Hasselt, 2010], prioritised replay [Schaul et al., 2016], dueling networks [Wang et al.,…”
Unsupervised Learning of Object Keypoints for Perception and Control (2019, methodology): “6, our approach is better than the state-of-the-art model-based SimPLe architecture [17] and model-free Rainbow architecture [11] on four out of five games.”
Towards a Definition of Disentangled Representations (2018, background): “Recent years have seen signiﬁcant progress in machine learning (ML), in particular in terms of supervised [44, 49, 73, 37] and reinforcement learning [60, 59, 45, 38, 29, 68].”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “[8] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver.”
BADGR: An Autonomous Self-Supervised Learning-Based Navigation System (2020, neutral): “reinforcement learning algorithms [43].”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, background): “It is telling that actor-critic or Q -learning algorithms advertised as oﬀ-policy still require large amounts of online interaction with the environment (Haarnoja et al., 2018; Hessel et al., 2018).”