Massively Parallel Methods for Deep Reinforcement Learning

By Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, A. Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charlie Beattie, Stig Petersen and 4 more. 2015

536 citations recorded by the source corpus.

We present the first massively distributed architecture for deep reinforcement learning. This architecture uses four main components: parallel actors that generate new behaviour; parallel learners that are trained from stored experience; a distributed neural network to represent the value function or behaviour policy; and a distributed store of experience. We used our architecture to implement the Deep Q-Network algorithm (DQN). Our distributed algorithm was applied to 49 games from Atari 2600 games from the Arcade Learning Environment, using identical hyperparameters. Our performance surpassed non-distributed DQN in 41 of the 49 games and also reduced the wall-time required to achieve these results by an order of magnitude on most games.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

TensorFlow: A system for large-scale machine learning (2016, neutral): “In recent years, machine learning has driven advances in many different ﬁelds [3, 5, 23, 24, 30, 27, 40, 45, 48, 50, 55, 68, 69, 73, 76].”
Emergence of Locomotion Behaviours in Rich Environments (2017, background): “on Atari games [9] and 3D navigation [2, 3].”
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems (2016, neutral): “com sequence prediction [47], move selection for Go [34], pedestrian detection [2], reinforcement learning [38], and other areas [17, 5].”
Mastering Atari, Go, chess and shogi by planning with a learned model (2019, methodology): “In Atari, we computed mean reward over 1000 episodes per game, limited to the standard 30 minutes or 108,000 frames per episode [27], using 50 simulations per move unless indicated otherwise.”
Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “In large-scale RL, decoupling acting from learning (Nair et al., 2015; Horgan et al., 2018; Espeholt et al., 2018) causes the experience to lag behind the latest agent policy.”
Distributed Prioritized Experience Replay (2018, methodology): “There are a growing number of examples to justify this optimism: effective use of greater computational resources has been a critical factor in the success of such algorithms as Gorila (Nair et al., 2015), A3C (Mnih et al., 2016), GPU Advantage Actor Critic (Babaeizadeh et al., 2017), Distributed PP”
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, methodology): “Examples include distributed A3C (Mnih et al., 2016), Gorila (Nair et al., 2015) and Ape-X (Horgan et al., 2018) with distributed replay, both distributed versions of Deep Q-Networks (Mnih et al., 2015).”
Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, background): “Asynchronous learning from parallel copies of the environment, as in A3C (Mnih et al. 2016), Gorila (Nair et al. 2015), or Evolution Strategies (Salimans et al. 2017), can be effective in speeding up learning, at least in terms of wall-clock time.”
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, methodology): “For example, scores as reported by Nair et al. (2015) are not comparable across starting states: although in a full game of Pong an agent can score 21 points, from a much later starting state this score is unachievable.”
Prioritized Experience Replay (2015, background): “Or, in a parallel system like the Gorila agent (Nair et al., 2015), it could guide resource allocation between a collection of concurrent but heterogeneous ‘actors’, each with different exploration hyperparameters.”
Deep Reinforcement Learning with Double Q-Learning (2015, methodology): “We obtained 100 starting points sampled for each game from a human expert’s trajectory, as proposed by Nair et al. (2015).”
Dueling Network Architectures for Deep Reinforcement Learning (2015, methodology): “Furthermore, as prioritization and the dueling architecture address very different aspects of the learning process, their combination is promising.”
Meta-Gradient Reinforcement Learning (2018, background): “The ﬁrst protocol is is “human starts” [Nair et al., 2015, Wang et al., 2016b, van Hasselt et al., 2016], which initialises episodes to a state that is randomly sampled from human play.”
Graying the black box: Understanding DQNs (2016, background): “…as a Markov Decision Process (MDP), i.e., to choose the discount factor, the amount of history frames that represent a state, and to choose between the various algorithms and architectures (Nair et al., 2015; Van Hasselt et al., 2015; Schaul et al., 2015; Wang et al., 2015; Bellemare et al., 2015).”
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation (2018, methodology): “We propose a parallel and distributed learning algorithm, inspired by [42], that can scale easily on cloud-based infrastructures.”
Deep Exploration via Bootstrapped DQN (2016, background): “Bootstrapped DQN is computationally tractable and also naturally scalable to massive parallel systems as per (Nair et al., 2015).”
Towards Characterizing Divergence in Deep Q-Learning (2019, methodology): “Data distribution-based methods include massive-scale experience collection, as in Gorila-DQN (Nair et al., 2015), Ape-X (Horgan et al., 2018), and R2D2 (Kapturowski et al., 2019), and methods for improved exploration, like entropy regularization (Haarnoja et al., 2018b).”
Distributional Reinforcement Learning with Quantile Regression (2017, background): “(10, 20, 30, 40, and 50) (Nair et al. 2015).”