Recurrent Experience Replay in Distributed Reinforcement Learning

By Steven Kapturowski, Georg Ostrovski, John Quan, R´emi Munos, Will Dabney. 2018

568 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “Major improvements have originated from scaling deep RL to a distributed setting across many machines (Mnih et al., 2016; Schulman et al., 2017; Barth-Maron et al., 2018; Horgan et al., 2018; Espeholt et al., 2018; Anonymous, 2019a).”
Mastering Atari, Go, chess and shogi by planning with a learned model (2019, methodology): “Game Random Human SimPLe [20] Ape-X [18] R2D2 [21] MuZero MuZero normalized”
Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “Note that the only previous agent “R2D2” that achieved a score beyond 400% required more than 3,000 million environment steps (see Kapturowski et al. (2019), page 14, Figure 9).”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “Experience replay has also been widely used in reinforcement learning (Mnih et al., 2015, 2013; Van Hasselt, Guez, & Silver, 2016; Lillicrap et al., 2016; Lin, 1992; Adam, Busoniu, & Babuska, 2011; Foerster et al., 2017; Kapturowski, Ostrovski, Dabney, Quan, & Munos, 2019; Atkinson et al., 2021).”
Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, neutral): “Our RL agents are based on the R2D2 architecture (Kapturowski et al., 2018) that are RNN-based DQN agents.”
A Framework for Data-Driven Robotics (2019, background): “Given the effectiveness of recurrent values functions [68], both Q and π are recurrent with hQt and ht representing the corresponding recurrent hidden states.”
Towards Characterizing Divergence in Deep Q-Learning (2019, methodology): “Data distribution-based methods include massive-scale experience collection, as in Gorila-DQN (Nair et al., 2015), Ape-X (Horgan et al., 2018), and R2D2 (Kapturowski et al., 2019), and methods for improved exploration, like entropy regularization (Haarnoja et al., 2018b).”
Regularized Behavior Value Estimation (2021, methodology): “Additionally, for the seekavoid _ arena _ 01 task we ran two fully trained snapshots of our R2D2 agents on the environment with different levels of noise ( (cid:15) = 0 , 0 .”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, methodology): “L G ] 1 J un 2 02 0 distributed systems in RL agent training (Mnih et al., 2016; Horgan et al., 2018a; Espeholt et al., 2018; Kapturowski et al., 2019).”
Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning (2019, methodology): “A combination of these techniques was ﬁrst explored by Kapturowski et al. (2019) in single agent environments such as Atari and DMLab-30.”
rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch (2019, methodology): “• Deep Q-Learning : DQN [1] + variants: Double [10], Dueling [11], Categorical [12], Rainbow [13] (minus Noisy Nets), Recurrent (R2D2-like) [6], including vector-valued epsilon-greedy (Ape-X-like) [14] ( coming soon : Implicit Quantile DQN [15]).”
Stabilizing Transformers for Reinforcement Learning (2019, background): “Yet, the RL literature is dominated by the use of LSTMs as the main mechanism for providing memory to the agent (Espeholt et al., 2018; Kapturowski et al., 2019; Mnih et al., 2016).”
A Generalist Agent (2022, methodology): “Data collection was done while training a R2D2 (Kapturowski et al., 2018) agent on each of the environments.”
Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions (2019, background): “In reinforcement learning (RL) [5], learning to play Atari was proposed in 2013 [6] and in recent years deep reinforcement learning (RL) algorithms reached and surpassed human play across a wide variety of Atari games [7–12].”
Optimizing agent behavior over long time scales by transporting value (2018, methodology): “Numbers are extracted from Table 4 in [6] and from the original A3C [7].”
Making Efficient Use of Demonstrations to Solve Hard Exploration Problems (2019, methodology): “This special case of R2D3 corresponds exactly to the R2D2 agent of Kapturowski et al. (2018), which itself extends DQN (Mnih et al., 2015) to partially observed environments by combining it with recurrence and the distributed training architecture of Ape-X DQN (Horgan et al., 2018).”
V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control (2019, methodology): “We note that the result for V-MPO at 10B environment frames across all levels matches the result for the Recurrent Replay Distributed DQN (R2D2) agent (Kapturowski et al., 2019) trained on individual levels for 10B environment steps per level.”
Transformers are Sample Efficient World Models (2022, methodology): “Before starting the imagination procedure from a given frame, we burn-in (Kapturowski et al., 2019) the 20 previous frames to initialize the hidden state.”
Scaling data-driven robotics with reward sketching and batch reinforcement learning (2019, background): “Given the effectiveness of recurrent value functions [36], both Q and π are recurrent with hQt and ht representing the corresponding recurrent hidden states.”
Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning (2020, methodology): “Billion-scale experiments with complex environments are now relatively commonplace (Horgan et al., 2018; Espeholt et al., 2018; Kapturowski et al., 2019), and the most advanced efforts consume trillions of environment transitions in a single training session (Berner et al.”