The Effects of Memory Replay in Reinforcement Learning

By Ruishan Liu, James Y. Zou. 2017

134 citations recorded by the source corpus.

Experience replay is a key technique behind many recent advances in deep reinforcement learning. Allowing the agent to learn from earlier memories can speed up learning and break undesirable temporal correlations. Despite its widespread application, very little is understood about the properties of experience replay. How does the amount of memory kept affect learning dynamics? Does it help to prioritize certain experiences? In this paper, we address these questions by formulating a dynamical systems ODE model of Q-learning with experience replay. We derive analytic solutions of the ODE for a simple setting. We show that even in this very simple setting, the amount of memory kept can substantially affect the agent’s performance–too much or too little memory both slow down learning. Moreover, we characterize regimes where prioritized replay harms the agent’s learning. We show that our analytic solutions have excellent agreement with experiments. Finally, we propose a simple algorithm for adaptively changing the memory buffer size which achieves consistently good empirical performance.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Management Science and Operations Research

Citations in the corpus

Continual Reinforcement Learning in 3D Non-stationary Environments (2019, background): “technique known as memory replay [34, 16, 23] is adopted, consisting in storing old observations in an external memory buffer to simulate an i.”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “While a number of prior works (Achiam et al., 2019; Fu et al., 2019; Liu and Zou, 2018) have provided a theoretical examination of various approximate dynamic programming (ADP) methods, prior work has not extensively studied the relationship between the data distribution induced by the latest value ”
A Deeper Look at Experience Replay (2017, background): “Liu and Zou (2017) did a theoretical study on the inﬂuence of the size of the replay buffer.”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, background): “In theory a replay buffer could prevent detachment, but in practice it would have to be large to prevent data about the abandoned frontier to not be purged before it becomes needed, and large replay buffers introduce their own optimization stability difficulties [21, 22].”
A Theoretical Analysis of Deep Q-Learning (2019, background): “Furthermore, recent work such as Schaul et al. (2016); Liu and Zou (2017); Zhang and Sutton (2017) study the effect of experience replay and propose various modifications.”
Revisiting Fundamentals of Experience Replay (2020, background): “Liu & Zou (2018) also study the effects of replay buffer size and minibatch size on learning performance.”
ReMindRAG: Low-Cost LLM-Guided Knowledge Graph Traversal for Efficient RAG (2025, unknown): “Following previous works regarding memory replay [29, 46], which refers to the recall of valuable past memories to facilitate current tasks, we design memory replay for efficient graph traversal.”