Efficient Lifelong Learning with A-GEM

By Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, Mohamed Elhoseiny. 2018

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “A-GEM [63] replaces the individual constraints with a global loss of all tasks to improve training efﬁciency.”
Learnability and Algorithm for Continual Learning (2023, methodology): “The replay methods are: iCaRL (Rebuffi et al., 2017), A-GEM (Chaudhry et al., 2019a), EEIL (Castro et al., 2018), GD (Lee et al., 2019) without external data, DER++ (Buzzega et al., 2020), HAL (Chaudhry et al., 2021), and MORE (Kim et al., 2022a).”
Progressive Prompts: Continual Learning for Language Models (2023, methodology): “• A-GEM (Chaudhry et al., 2018): save examples from the past tasks and restrict the gradients used to update the model on new tasks based on the retrieved examples.”
RMM: Reinforced Memory Management for Class-Incremental Learning (2023, background): “Recent papers are either task-incremental learning—each phase corresponds to a task (dataset) that contains new data of all seen classes [7, 10, 19, 29, 43, 47, 57], or classincremental learning (CIL)—each phase contains data of a new set of classes, i.”
Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer (2022, methodology): “, [3, 6, 25]) aim to update the model with minimal interference introduced to old tasks.”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, neutral): “Indeed, due to the limited size of the training set, competitive performance can only be achieved if each task can beneﬁt from the initialization [20, 88].”
Continual Learning by Modeling Intra-Class Variation (2022, methodology): “Similar to our work, gradient-based regularization methods (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018; Saha et al., 2021) try to restrict the update directions in a common space where gradients from old classes and new classes have the largest inner product.”
Class Gradient Projection For Continual Learning (2022, methodology): “Concretely, thememory-based approaches include reservoir sampling (ER_Res) [8], Averaged GEM (A-GEM) [7], and Orthogonal Weight Modulation (OWM) [44].”
Disentangling Transfer in Continual Reinforcement Learning (2022, neutral): “[6, 7]; see also CL survey papers [8, 17, 32].”
Repeated Augmented Rehearsal: A Simple but Strong Baseline for Online Continual Learning (2022, methodology): “We also compare it with other continual learning methods, including the regularization-based method LWF [Li and Hoiem, 2017] and the constrained optimization-based method A-GEM [Chaudhry et al., 2018].”
Exploring Example Influence in Continual Learning (2022, neutral): “Rehearsal-based methods [29, 7, 22, 6, 1, 2, 28, 3, 25] tackle the challenge of SP dilemma by retaining a subset of old tasks in a stored memory buffer with bounded resources.”
Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, neutral): “Many works mitigate catastrophic forgetting during the learning process [37,14,48,68,31,42,20,2,10,3,64].”
DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning (2022, background): “Rehearsal-based approaches [5, 26, 34, 36] replay examples from previous tasks either by storing samples to an external memory buffer or learning a generative model.”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “Methods such as GEM [34], A-GEM [10], OGD [14] and NSCL [50] fall under this category.”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “For instance, memory-based methods alleviate forgetting by either replaying a small set of examples from past tasks saved in a memory [57, 55, 46, 7, 23, 28, 9] or replaying synthesized data from previous tasks using generative models [60, 63].”