Revisiting Fundamentals of Experience Replay

By William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, Will Dabney. International Conference on Machine Learning, 2020

318 citations recorded by the source corpus.

Experience replay is central to off-policy algorithms in deep reinforcement learning (RL), but there remain significant gaps in our understanding. We therefore present a systematic and extensive analysis of experience replay in Q-learning methods, focusing on two fundamental properties: the replay capacity and the ratio of learning updates to experience collected (replay ratio). Our additive and ablative studies upend conventional wisdom around experience replay -- greater capacity is found to substantially increase the performance of certain algorithms, while leaving others unaffected. Counterintuitively we show that theoretically ungrounded, uncorrected n-step returns are uniquely beneficial while other techniques confer limited benefit for sifting through larger memory. Separately, by directly controlling the replay ratio we contextualize previous observations in the literature and empirically measure its importance across a variety of deep RL algorithms. Finally, we conclude by testing a set of hypotheses on the nature of these performance benefits.

Subfields: Computer Networks and Communications, Electrical and Electronic Engineering, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/517498898024107a75f5a4ea84f4129e300af603

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Electrical and Electronic Engineering

In Management Science and Operations Research

Citations in the corpus

Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning (2021, background): “Replay Buffer Size We hypothesize that a larger replay buffer plays an important role in circum-venting the catastrophic forgetting problem [Fedus et al., 2020].”
Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation (2021, background): “Furthermore, the sufﬁciently large memory may increase the degree of off-policyness of transitions in the buffer [28], violating the requirement of current state-of-the-art algorithms that the data should be close to the on-policy distribution, even for off-policy algorithms such as deep Q network…”
Reverb: A Framework For Experience Replay (2021, methodology): “This ratio is commonly used to set the number of gradient updates that are taken per data element, which can signiﬁcantly impact RL algorithm performance (Fedus et al., 2020; Fu et al., 2019; Hoffman et al., 2020).”
Model-augmented Prioritized Experience Replay (2022, background): “The effect of increasing capacity and downweighting old transition in the buffer has been also studied (Fedus et al., 2020).”
The Importance of Pessimism in Fixed-Dataset Policy Optimization (2020, background): “ese algorithms are oen very sensitive to hyperparemters; in particular, they perform poorly when the number of learning steps per interaction is large [6].”
Transient Non-stationarity and Generalisation in Deep Reinforcement Learning (2020, background): “Furthermore, in contrast to Supervised Learning (SL), simply re-training the agent from scratch once the data-distribution has changed is infeasible in RL as current state of the art algorithms require data close to the on-policy distribution, even for off-policy algorithms like Q-learning (Fedus et”
Regret Minimization Experience Replay in Off-Policy Reinforcement Learning (2021, unknown): “Other works focus on buffer capacity [20, 21].”
The Primacy Bias in Deep Reinforcement Learning (2022, background): “Even though there exist works studying in detail the pathological behaviors of TD learning (Bengio et al., 2020) and this is not the main scope of our work, resets naturally address the outlined failures.”