A Deeper Look at Experience Replay

By Shangtong Zhang, Richard S. Sutton. 2017

323 citations recorded by the source corpus.

Recently experience replay is widely used in various deep reinforcement learning (RL) algorithms, in this paper we rethink the utility of experience replay. It introduces a new hyper-parameter, the memory buffer size, which needs carefully tuning. However unfortunately the importance of this new hyper-parameter has been underestimated in the community for a long time. In this paper we did a systematic empirical study of experience replay under various function representations. We showcase that a large replay buffer can significantly hurt the performance. Moreover, we propose a simple O(1) method to remedy the negative influence of a large replay buffer. We showcase its utility in both simple grid world and challenging domains like Atari games.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Off-Policy Actor-Critic with Shared Experience Replay (2019, methodology): “Mixing online data with replay data has also been argued for by Zhang & Sutton (2017), as a heuristic way of reducing the sensitivity of reinforcement learning algorithms to the size of the replay memory.”
Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “It has been observed that large experience replay (Lin, 1992) can be detrimental to performance (de Bruin et al., 2015; Zhang & Sutton, 2017) and the diversity of states in the buffer is an important factor for performance (de Bruin et al., 2016).”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, methodology): “We perform some preliminary experiments on a sequential version of symmetric noisy MNIST and CIFAR-10 [53, 84] using experience replay with the conventional reservoir sampling technique [69, 94].”
Continual Reinforcement Learning with Multi-Timescale Replay (2020, background): “It was designed with the following three motivating factors: • Several of the previously mentioned replay methods use just two timescales of memory in order to strike a balance between new and old experiences [Isele and Cosgun, 2018, Rolnick et al., 2019, Zhang and Sutton, 2017].”
Learning to Learn without Forgetting By Maximizing Transfer and Minimizing Interference (2018, methodology): “This procedure closely follows recent enhancements discussed in (Zhang & Sutton, 2017; Riemer et al., 2017b;a) The sample function randomly samples k − 1 examples from the memory buffer M and interleaves them with the current example to form a single size k batch.”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “[52] Shangtong Zhang and Richard S Sutton.”
Diagnosing Bottlenecks in Deep Q-learning Algorithms (2019, methodology): “In deep RL, several methods have been developed to prevent instabilities in Q-Learning, such as prioritized replay (Schaul et al., 2015), and mixing replay buffer with on-policy data (Hausknecht & Stone, 2016; Zhang & Sutton, 2017) have been found to be beneﬁcial.”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, background): “ctice it would have to be large to prevent data about the abandoned frontier to not be purged before it becomes needed, and large replay buffers introduce their own optimization stability difﬁculties [21, 22]. The Go-Explore algorithm addresses detachment by explicitly storing an archive of promisin”
Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2 (2022, background): “Others have explored sophisticated strategies for experience replay in DQN (e.g. Isele & Cosgun, 2018; Riemer et al., 2019; Wang et al., 2021; Caccia et al., 2020; Zhang & Sutton, 2017; Schaul et al., 2015) in lifelong RL, but these are not generative in nature.”
A Theoretical Analysis of Deep Q-Learning (2019, background): “Furthermore, recent work such as Schaul et al. (2016); Liu and Zou (2017); Zhang and Sutton (2017) study the effect of experience replay and propose various modifications.”
An Optimistic Perspective on Offline Reinforcement Learning (2019, background): “Furthermore, Zhang & Sutton (2017) assert that a large replay buffer can even hurt the performance of off-policy algorithms due to its “off-policyness”.”
Time Limits in Reinforcement Learning (2017, methodology): “Finally, we demonstrate that the negative impact of large experience replay buffers shown by Zhang & Sutton (2017) can often be vastly reduced if timeout terminations are properly handled.”
Revisiting Fundamentals of Experience Replay (2020, background): “In earlier work, Zhang & Sutton (2017) studied the effects of increasing the size of the replay buffer of DQN.”
Model-augmented Prioritized Experience Replay (2022, background): “Wang et al. (2015); Zhang & Sutton (2015) also stuided the importance of utilizing the newest experiences for RL’s learning.”
Simplifying Deep Reinforcement Learning via Self-Supervision (2021, background): “While many studies have been focused on advancing experience replay (Andrychowicz et al. 2017; Pan et al. 2018; Zhang and Sutton 2017; Novati and Koumoutsakos 2018; Zha et al. 2019b), the main goal of the prior work is to accelerate or stabilize the training of value functions.”