Overcoming Catastrophic Forgetting by Incremental Moment Matching

By Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, Byoung-Tak Zhang. 2017

738 citations recorded by the source corpus.

Catastrophic forgetting is a problem of neural networks that loses the information of the first task after training the second task. Here, we propose a method, i.e. incremental moment matching (IMM), to resolve this problem. IMM incrementally matches the moment of the posterior distribution of the neural network which is trained on the first and the second task, respectively. To make the search space of posterior parameter smooth, the IMM procedure is complemented by various transfer learning techniques including weight transfer, L2-norm of the old and the new parameter, and a variant of dropout with the old parameter. We analyze our approach on a variety of datasets including the MNIST, CIFAR-10, Caltech-UCSD-Birds, and Lifelog datasets. The experimental results show that IMM achieves state-of-the-art performance by balancing the information between an old and a new network.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “…of the difference between parameters for previously learned tasks during parameter updates [Hashimoto et al., 2016], incrementally matching modes [Lee et al., 2017], rehearsing on old tasks [Robins, 1995], using adaptive memory buffers [Gepperth and Karaoguz, 2016], finding task-specific paths…”
Toward Understanding Catastrophic Forgetting in Continual Learning (2019, background): “Other algorithmic ideas to prevent catastrophic forgetting include moment matching [25], learning without forgetting [26], and deep generative replay [13, 19, 42].”
Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines (2018, background): “This evaluation scenario is widely adopted [3, 4, 5, 6, 7, 8, 9], despite criticism that it is less challenging in terms of forgetting [10].”
Continual Learning with Deep Generative Replay (2017, background): “A branch of works assumes a particular situation where access to previous data is limited to the current task[12, 18, 20].”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “IMM [111] is an early attempt that incrementally matches the moment of the posterior distributions for old and new tasks, i.e., a weighted average of their solutions.”
Variational Continual Learning (2017, background): “One approach trains individual models on each task and then carries out a second stage of training to combine them (Lee et al., 2017).”
Overcoming catastrophic forgetting with hard attention to the task (2018, methodology): “We match the base network architecture to the one used by Lee et al. (2017).”
Lifelong Learning with Dynamically Expandable Networks (2017, neutral): “Recently, lifelong learning is studied in deep learning frameworks; since lifelong learning of a deep network can be straightforwardly done by simple re-training, the primary focus of research is on overcoming catastrophic forgetting (Kirkpatrick et al., 2017; Rusu et al., 2016; Zenke et al., 2017; ”
Continual Learning via Sequential Function-Space Variational Inference (2023, methodology): “Elastic weight consolidation ( EWC ; Kirkpatrick et al., 2017) and its successors (Chaudhry et al., 2018; Lee et al., 2017; Liu et al., 2018; Schwarz et al., 2018) use a Fisher information matrix computed on each task.”
Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer (2022, background): “, [12, 1, 15, 18]) add additional regularization in the loss function to penalize the model change of the old tasks when learning the new task.”
DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning (2022, methodology): “Regularization-based methods aim to penalize the update either in function space [8, 24] or in parameter space [2, 9, 18, 20, 22, 23, 32, 38, 45].”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “In addition, there exist other kinds of methods like gradient-based methods (e.g., [56, 73, 16, 58, 61, 64]), regularization-based ones (e.g. [32, 74, 2, 1]), and expansion-based ones (e.g. [69, 34, 25]).”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “Moreover, with new tasks occurring, the number of previous tasks increases, resulting in a greater impact of previous tasks on the final performance than the current task [19, 21, 58].”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, methodology): “(3) IMM (Lee et al. 2017) combines the sequentially trained independent models for different tasks to perform all the tasks in the sequence.”
Continual Learning with Guarantees via Weight Interval Constraints (2022, methodology): “Although several methods effective at reducing forgetting when learning new tasks were proposed (Kirkpatrick et al., 2017; Lee et al., 2017; Li & Hoiem, 2017; Lopez-Paz & Ranzato, 2017; Shin et al., 2017; Zenke et al., 2017; Aljundi et al., 2018; Masse et al., 2018; Liu et al., 2018; Rolnick et…”
Meta-attention for ViT-backed Continual Learning (2022, methodology): “The regularizationbased methods [21, 23, 26, 27, 34, 50, 52] utilize the knowledge distillation technique [15], special regularization terms to avoid catastrophic forgetting.”
Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation (2022, background): “Lee et al. (2017) proposed drop-transfer, which is a variant of dropout that drops the weight vector of turned off nodes to the weight learned on the previous task instead of a zero vector.”
TRGP: Trust Region Gradient Projection for Continual Learning (2022, methodology): “This category of methods (e.g., (Kirkpatrick et al., 2017; Lee et al., 2017; Chaudhry et al., 2018a; Dhar et al., 2019; Ritter et al., 2018; Schwarz et al., 2018; Zenke et al., 2017)) protect the old tasks by adding regularization terms in the loss function to penalize the model change on their…”
Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks (2021, background): “, 2018), PackNet (Mallya and Lazebnik, 2018), Diffusion-based Neuromodulation (Velez and Clune, 2017), IMM (Lee et al., 2017), Expandable Networks (Yoon et al.”