Gradient Episodic Memory for Continuum Learning

By David Lopez-Paz, Marc'Aurelio Ranzato. 2017

63 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Meta & Continual Learning

Citations in the corpus

Continuous Learning in Single-Incremental-Task Scenarios (2018, methodology): “In the rehearsal category, Gradient Episodic Memory (GEM) [20] is an interesting approach using a fixed memory to store a subset of old patterns: it is aimed not only at controlling forgetting but also at improving accuracy on previous tasks while learning the subsequent ones (a phenomenon known as ”
Distill and Replay for Continual Language Learning (2020, background): “We find that fine-tuning, EWC, MAS and GEM are not robust to the variation of task order when learning different types of tasks in a stream.”
Continual Learning in Low-rank Orthogonal Subspaces (2020, methodology): “AGEM [Chaudhry et al., 2019a] is a memory-based method similar to [Lopez-Paz and Ranzato, 2017] that uses episodic memory as an optimization constraint to reduce forgetting on previous tasks.”
Using Hindsight to Anchor Past Knowledge in Continual Learning (2019, methodology): “Third, the memory-based methods (Lopez-Paz and Ranzato 2017; Hayes, Cahill, and Kanan 2018; Isele and Cosgun 2018; Riemer et al. 2019; Chaudhry et al. 2019a) store a few examples from past tasks in an “episodic memory”, to be revisited when training for a new task.”
Episodic Memory in Lifelong Language Learning (2019, background): “Recent methods have shown that these approaches can be combined—e.g., by deﬁning optimization constraints using samples from the episodic memory (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019).”
Efficient Lifelong Learning with A-GEM (2018, methodology): “LCA 0 is the average 0-shot performance, the same as forward transfer in Lopez-Paz & Ranzato (2017).”
Learning to Learn without Forgetting By Maximizing Transfer and Minimizing Interference (2018, methodology): “Namely, we follow a commonly used setting to define non-stationary conditions for continual learning, dubbed locally i.i.d by (Lopez-Paz & Ranzato, 2017), where the agent learns over a sequence of separate stationary distributions one after another.”
Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence (2018, methodology): “However, in [14], FT is quantified as the gain in accuracy compared to the random guess (not a measure of intransigence) which is complementary to our approach.”
Scalable Recollections for Continual Lifelong Learning (2017, methodology): “MNIST-Rotations: (Lopez-Paz and Ranzato 2017) A dataset with 20 tasks including 1,000 training examples for each task.”
BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning (2020, methodology): “For example, gradient episodic memory approach penalizes the gradient on the current task so that it does not increase the loss of examples in the memory buffer (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018).”
Continual Learning with Tiny Episodic Memories (2019, methodology): “To validate this hypothesis, we conducted experiments on the MNIST Rotations dataset [Lopez-Paz and Ranzato, 2017], where each task has digits rotated by a certain degree.”
Latent Replay for Real-Time Continual Learning (2019, background): “proved to be an effective approach to contrast forgetting in continual learning scenarios [17], [18], [19], [20].”
On Tiny Episodic Memories in Continual Learning (2019, methodology): “…seeing a surge of interest in the research community with several methods proposed to tackle catastrophic forgetting [Rebuffi et al., 2017; Kirkpatrick et al., 2016; Zenke et al., 2017; Lee et al., 2017a; Aljundi et al., 2018; Lopez-Paz and Ranzato, 2017; Lee et al., 2017b; Chaudhry et al., 2019].”
SPACE: Structured Compression and Sharing of Representational Space for Continual Learning (2020, methodology): “The first set of methods involves a form of rehearsal over older tasks while training the current task, either with saved examples [30, 29, 23] or with synthetic data generated by a generator trained over older tasks [34].”
Efficient Lifelong Learning with A-GEM (2018, methodology): “LCA 0 is the average 0-shot performance, the same as forward transfer in Lopez-Paz & Ranzato (2017).”
Class Incremental Learning for Video Action Classification (2021, background): “…to overcome catastrophic forgetting: (1) To regularize the important parameters of the old model when training new data [9, 10, 11, 12], (2) To retain some old samples for knowledge distillation technology [13, 14, 15, 16, 17], (3) To generate old samples based on generative networks [18, 19, 20].”