Distributed Prioritized Experience Replay

By Dan Horgan, John Quan, David M. Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, David Silver. 2018

833 citations recorded by the source corpus.

We propose a distributed architecture for deep reinforcement learning at scale, that enables agents to learn effectively from orders of magnitude more data than previously possible. The algorithm decouples acting from learning: the actors interact with their own instances of the environment by selecting actions according to a shared neural network, and accumulate the resulting experience in a shared experience replay memory; the learner replays samples of experience and updates the neural network. The architecture relies on prioritized experience replay to focus only on the most significant data generated by the actors. Our architecture substantially improves the state of the art on the Arcade Learning Environment, achieving better final performance in a fraction of the wall-clock training time.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Reinforcement Learning

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “Major improvements have originated from scaling deep RL to a distributed setting across many machines (Mnih et al., 2016; Schulman et al., 2017; Barth-Maron et al., 2018; Horgan et al., 2018; Espeholt et al., 2018; Anonymous, 2019a).”
Mastering Atari, Go, chess and shogi by planning with a learned model (2019, methodology): “Game Random Human SimPLe [20] Ape-X [18] R2D2 [21] MuZero MuZero normalized”
Addressing Function Approximation Error in Actor-Critic Methods (2018, methodology): “Orthogonal to our approach, recent improvements to DDPG include distributed methods (Popov et al., 2017), along with multi-step returns and prioritized experience replay (Schaul et al., 2016; Horgan et al., 2018), and distributional methods (Bellemare et al., 2017; Barth-Maron et al., 2018).”
Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “Prioritized experience replay has the potential to provide more efﬁcient learning compared to uniform experience replay (Schaul et al., 2015; Horgan et al., 2018).”
Observe and Look Further: Achieving Consistent Performance on Atari (2018, methodology): “Note that the scores reported in [8] were obtained by running 360 actors.”
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, methodology): “Examples include distributed A3C (Mnih et al., 2016), Gorila (Nair et al., 2015) and Ape-X (Horgan et al., 2018) with distributed replay, both distributed versions of Deep Q-Networks (Mnih et al., 2015).”
RUDDER: Return Decomposition for Delayed Rewards (2018, methodology): “owling and Venture faster than other methods. Normalized human-percentage scores during training for Bowling (left) and for Venture (right), where learning curves are taken from previous publications [39, 50]. RUDDER sets a new state-of-the-art for Bowling. RUDDER Implementation for Atari Games. We ”
Dota 2 with Large Scale Deep Reinforcement Learning (2019, methodology): “If data is reused too often, it can lead to overfitting on the reused data[18].”
Unsupervised Learning of Object Keypoints for Perception and Control (2019, methodology): “How do we learn skills using object keypoints for efficient exploration? We use a distributed off-policy learner similar to [13] using 128 actors and 4 GPUs.”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, methodology): “Prioritized sampling has been used previously in ADP methods to instead prioritize transitions with higher Bellman error (Schaul et al., 2015; Hessel et al., 2018; Hou et al., 2017; Horgan et al., 2018).”
Deep Reinforcement Learning and the Deadly Triad (2018, background): “Multi-step returns have already been shown to be beneﬁcial for performance in certain variations of DQN (Hessel et al., 2018; Horgan et al., 2018).”
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation (2018, background): “Similar to [45], our distributed replay buffer is shared across all agents, although we do not use prioritized replay.”
Towards Characterizing Divergence in Deep Q-Learning (2019, methodology): “Ape-X DQfD (Pohlen et al., 2018) uses an auxilliary temporal consistency (TC) loss to prevent the Q -values of target state-action pairs from changing; ablation analysis suggested that the TC loss was critical to performance.”
Exploration by Random Network Distillation (2018, background): “In Table 1 (Fortunato et al., 2017) [2] (Bellemare et al., 2016) [3] (Horgan et al., 2018) 3.7 Q UALITATIVE A NALYSIS : D ANCING WITH SKULLS By observing the RND agent, we notice that frequently once it obtains all the extrinsic rewards that it knows how to obtain reliably (as judged by the…”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “L G ] 1 J un 2 02 0 distributed systems in RL agent training (Mnih et al., 2016; Horgan et al., 2018a; Espeholt et al., 2018; Kapturowski et al., 2019).”
Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning (2019, methodology): “We also employ a distributed training architecture similar to the one proposed by Horgan et al. (2018) where a number of different actors with their own exploration rates collect experiences in parallel and feed them into a central replay buffer.”
Google Research Football: A Novel Reinforcement Learning Environment (2019, methodology): “In particular, we chose Ape-X DQN (Horgan et al. 2018), a highly scalable version of DQN.”
rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch (2019, background): “• Deep Q-Learning : DQN [1] + variants: Double [10], Dueling [11], Categorical [12], Rainbow [13] (minus Noisy Nets), Recurrent (R2D2-like) [6], including vector-valued epsilon-greedy (Ape-X-like) [14] ( coming soon : Implicit Quantile DQN [15]).”
Distributed Distributional Deterministic Policy Gradients (2018, methodology): “We implement this procedure using the ApeX framework (Horgan et al., 2018) and refer the reader there for more details.”