Experience Replay for Continual Learning

By David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, Greg Wayne. Neural Information Processing Systems, 2018

1,717 citations recorded by the source corpus.

Continual learning is the problem of learning new tasks or knowledge while protecting old knowledge and ideally generalizing from old experience to learn new tasks faster. Neural networks trained by stochastic gradient descent often degrade on old tasks when trained successively on new tasks with different data distributions. This phenomenon, referred to as catastrophic forgetting, is considered a major hurdle to learning with non-stationary data or sequences of new tasks, and prevents networks from continually accumulating knowledge and skills. We examine this issue in the context of reinforcement learning, in a setting where an agent is exposed to tasks in a sequence. Unlike most other work, we do not provide an explicit indication to the model of task boundaries, which is the most general circumstance for a learning agent exposed to continuous experience. While various methods to counteract catastrophic forgetting have recently been proposed, we explore a straightforward, general, and seemingly overlooked solution - that of using experience replay buffers for all past events - with a mixture of on- and off-policy learning, leveraging behavioral cloning. We show that this strategy can still learn new tasks quickly yet can substantially reduce catastrophic forgetting in both Atari and DMLab domains, even matching the performance of methods that require task identities. When buffer storage is constrained, we confirm that a simple mechanism for randomly discarding data allows a limited size buffer to perform almost as well as an unbounded one.

Subfields: Artificial Intelligence, Computer Science Applications, Management Science and Operations Research, Reinforcement Learning, Meta & Continual Learning

View Paper: https://arxiv.org/abs/1811.11682

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yoshua Bengio
  2. Sergey Levine
  3. Mohit Bansal
  4. Dacheng Tao
  5. Jianfeng Gao
  6. Quoc V. Le
  7. Trevor Darrell
  8. Ruslan Salakhutdinov

In Computer Science Applications

  1. Rahaf Aljundi
  2. Fei Sha
  3. Igor Mordatch
  4. Massimo Caccia
  5. Xiang Ren
  6. Xisen Jin
  7. Junyi Du
  8. Lucas Caccia

In Management Science and Operations Research

  1. Sergey Levine
  2. Chelsea Finn
  3. Pieter Abbeel
  4. Yoshua Bengio
  5. Josh Tenenbaum
  6. Maosong Sun
  7. Behnam Neyshabur
  8. Zhiyuan Liu

In Reinforcement Learning

  1. Sergey Levine
  2. Chelsea Finn
  3. Pieter Abbeel
  4. Joelle Pineau
  5. Shimon Whiteson
  6. Abhinav Kumar Gupta
  7. Igor Mordatch
  8. Razvan Pascanu

In Meta & Continual Learning

  1. Tinne Tuytelaars
  2. Jorge Armando Mendez Mendez
  3. Zixuan Ke
  4. Eric Eaton
  5. Bing Liu
  6. Han-Jia Ye
  7. Fei Ye
  8. Adrian G. Bors

Citations in the corpus