Learning to Learn without Forgetting By Maximizing Transfer and Minimizing Interference
By Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, G. Tesauro. 2018
934 citations recorded by the source corpus.
Lack of performance when it comes to continual learning over non-stationary distributions of data remains a major challenge in scaling neural network learning to more human realistic settings. In this work we propose a new conceptualization of the continual learning problem in terms of a temporally symmetric trade-off between transfer and interference that can be optimized by enforcing gradient alignment across examples. We then propose a new algorithm, Meta-Experience Replay (MER), that directly exploits this view by combining experience replay with optimization based meta-learning. This method learns parameters that make interference based on future gradients less likely and transfer based on future gradients more likely. We conduct experiments across continual lifelong supervised learning benchmarks and non-stationary reinforcement learning environments demonstrating that our approach consistently outperforms recently proposed baselines for continual learning. Our experiments show that the gap between the performance of MER and baseline algorithms grows both as the environment gets more non-stationary and as the fraction of the total experiences stored gets smaller.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Dacheng Tao
- Sergey Levine
- Yoshua Bengio
- Bernt Schiele
- Mohit Bansal
- Jun Zhu
- Han-Jia Ye
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Bernt Schiele
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Yi Yang
- Andrea Vedaldi
In Representation & Self-Supervised Learning
- Han-Jia Ye
- De-chuan Zhan
- Dacheng Tao
- Joost van de Weijer
- Bernt Schiele
- Sergey Levine
- Tinne Tuytelaars
- Trevor Darrell
In Meta & Continual Learning
- Han-Jia Ye
- Liyuan Wang
- Jun Zhu
- De-chuan Zhan
- Xingxing Zhang
- Jorge Armando Mendez Mendez
- Eric Eaton
- Joost van de Weijer
Citations in the corpus
- A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Forexample, MER[131]alignstheirgradientdirections,whileiTAML[184] appliesameta-updatingruletokeeptheminbalancewitheach other.”
- Incorporating neuro-inspired adaptability for continual learning in artificial intelligence (2023, methodology): “Finally, we evaluate CAF under the setting of task-incremental learning [58] and compare it to a range of regularization-based methods [7–9, 44, 46, 59, 60].”
- The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “Based on prior works (Vitter, 1985; Lopez-Paz & Ranzato, 2017; Rebuffi et al., 2017; Riemer et al., 2019), Chaudhry et al.”
- RMM: Reinforced Memory Management for Class-Incremental Learning (2023, background): “…papers are either task-incremental learning—each phase corresponds to a task (dataset) that contains new data of all seen classes [7, 10, 19, 29, 43, 47, 57], or class-incremental learning (CIL)—each phase contains data of a new set of classes, i.e., classes are unseen [4, 6, 18, 25, 31, 37,…”
- Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer (2022, background): “In stark contrast, the task gradient correlation characterization in [3, 22] involves the gradient evaluation of old tasks with respect to the current model weight, and hence requires the data of old tasks when learning the new task.”
- On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, neutral): “In this respect, different approaches establish different regularization strategies on top of the retained examples [63, 49], as well as which kind of information to store (e.”
- Exploring Example Influence in Continual Learning (2022, neutral): “To suppress the catastrophic forgetting, the rehearsal-based CL [30, 22, 32, 6, 16] builds a small size memory bufferMt sampled from Dtrn t for each task (i.”
- Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, neutral): “Many works mitigate catastrophic forgetting during the learning process [37,14,48,68,31,42,20,2,10,3,64].”
- DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning (2022, neutral): “Method Task-IL Class-IL 0 LwF [24] 92.16 - EWC [18] 77.44 - OSLA [39] 81.03 - DLCFT (Ours) 95.79 - 500 ER [38] 79.14 43.52 DER [4] 91.47 58.07 DER++ [4] 91.56 53.29 DLCFT (Ours) - 59.98”
- Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “MER [43], an algorithm-based method is related to our high-level objective.”
- S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “In addition, there exist other kinds of methods like gradient-based methods (e.g., [56, 73, 16, 58, 61, 64]), regularization-based ones (e.g. [32, 74, 2, 1]), and expansion-based ones (e.g. [69, 34, 25]).”
- Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “These method can be roughly divided into four categories: architecture-based methods expand the network or allocate new neurons for new tasks [13,28,42,60]; replayedbased methods interleave old data with current data by storing historical data in a buffer or generating virtual old data [4,8,12,39,40”
- Online Continual Learning with Contrastive Vision Transformer (2022, methodology): “Continual learning (also known as lifelong learning or incremental learning) [62,63,79,75,80,95,18] aims to solve this problem by maintaining and accumulating the acquired knowledge over time from a stream of non-stationary data.”
- Incremental Task Learning with Incremental Rank Updates (2022, methodology): “Incremental task learning is a subcategory of continual learning or lifelong learning approaches aim to address the problem of catastrophic forgetting by adapting the network or training process to learn new tasks without forgetting the previously learned ones [23,16,3,2,4,6,28,29,12].”
- Improving Task-free Continual Learning by Distributionally Robust Memory Evolution (2022, background): “The memory buffer M is updated by reservoir sampling (RS), similar to (Riemer et al., 2019).”
- CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One (2022, positive): “(ER) of old training samples (20 images per class) [37].”
- Continual Learning through Retrieval and Imagination (2022, methodology): “Unfortunately, most of the exiting methods fail to satisfy all these guidelines mentioned above (Chaudhry et al. 2021; Wang et al. 2020; Lopez-Paz et al. 2017; Wei et al. 2021; Riemer et al. 2019; Kong et al. 2021; Schwarz et al. 2018; Zhu et al. 2020).”
- Transfer without Forgetting (2022, background): “MER [52] combines replay and meta-learning [21,44] to maximize transfer from the past while minimizing interference.”