Less-forgetting Learning in Deep Neural Networks

By Heechul Jung, Jeongwoo Ju, Minju Jung, Junmo Kim. 2016

242 citations recorded by the source corpus.

A catastrophic forgetting problem makes deep neural networks forget the previously learned information, when learning data collected in new environments, such as by different sensors or in different light conditions. This paper presents a new method for alleviating the catastrophic forgetting problem. Unlike previous research, our method does not use any information from the source domain. Surprisingly, our method is very effective to forget less of the information in the source domain, and we show the effectiveness of our method using several experiments. Furthermore, we observed that the forgetting problem occurs between mini-batches when performing general training processes using stochastic gradient descent methods, and this problem is one of the factors that degrades generalization performance of the network. We also try to solve this problem using the proposed method. Finally, we show our less-forgetting learning method is also helpful to improve the performance of deep neural networks in terms of recognition rates.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

Knowledge Flow: Improve Upon Your Teachers (2019, methodology): “Similar techniques have been developed by Furlanello et al. (2016); Jung et al. (2016).”
Continual and Multi-Task Architecture Search (2019, methodology): “Some approaches addressed this by adding regularization to penalize functional or shared parameters’ change and learning rates (Razavian et al., 2014; Li and Hoiem, 2017; Hinton et al., 2015; Jung et al., 2016; Kirk-patrick et al., 2017; Donahue et al., 2014; Yosinski et al., 2014).”
Overcoming catastrophic forgetting with hard attention to the task (2018, methodology): “ (Goodfellow et al.,2014), SGD freezing all layers except the last one (SGD-F), EWC, IMM (Mean and Mode variants), learning without forgetting (LWF;Li & Hoiem,2017), less-forgetting learning (LFL;Jung et al., 2016), PathNet, and PNNs. To ﬁnd the best hyperparameter combination for each approach,”
Gradient Episodic Memory for Continual Learning (2017, background): “Other approaches are instead based on “episodic” memory [Jung et al., 2016, Li and Hoiem, 2016, Rannen Triki et al., 2017, Rebufﬁ et al., 2017], where examples from previous tasks are stored and replayed to maintain predictions invariant by means of distillation [Hinton et al., 2015].”
A Theoretical Study on Solving Continual Learning (2022, methodology): “Using regularization [12, 13] to minimize changes to model parameters learned from previous tasks is a popular approach [14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26].”
Task-Free Continual Learning via Online Discrepancy Distance Learning (2022, neutral): “Unfortunately, modern deep learning models suffer from a degenerated performance on past data after learning novel knowledge, a phenomenon called catastrophic forgetting [13].”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, neutral): “Regularization-based methods can be divided into two classes: one is to distill knowledge from the previous model which is trained on the previous tasks [14, 15, 23, 36, 52, 59]; another is to explicitly use a regularizer to penalize the update of important parameters of previous tasks, preventing t”
Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation (2022, background): “Knowledge distillation approaches [21, 26] focus on minimizing the divergences from the representations of the previous models while effectively adapting to new tasks.”
Meta-attention for ViT-backed Continual Learning (2022, methodology): “The regularizationbased methods [21, 23, 26, 27, 34, 50, 52] utilize the knowledge distillation technique [15], special regularization terms to avoid catastrophic forgetting.”
Online Continual Learning Via Candidates Voting (2021, methodology): “Representative methods include freezing part of layers [21, 23] and using distillation loss or its variants [26, 13, 6, 35, 18, 19, 25, 14].”
Layerwise Optimization by Gradient Decomposition for Continual Learning (2021, positive): “The latter approaches, such as LWF [29], LFL [20], LWM [9] and BLD [11], leveraged knowledge distillation [16, 58] among consecutive tasks.”
Learning latent representations across multiple data domains using Lifelong VAEGAN (2020, background): “Regularization approaches in order to alleviate catastrophic forgetting add an auxiliary term that penalizes changes in the weights when the model is trained on a new task [14,26,29,33,38,48,54,60].”
Incremental Learning in Online Scenario (2020, methodology): “In [10], the last fully connected layer is freezed which discourages change of shared parameters in the feature extraction layers.”
Continual Learning with Adaptive Weights (CLAW) (2019, methodology): “The l2 distance between the top hidden activations of the old and new tasks is used for regularisation in (Jung et al., 2016).”
Uncertainty-guided Continual Learning with Bayesian Neural Networks (2019, methodology): “It is interesting to note that in this setup, some baselines (PathNet, LWF, and LFL) do not perform better than the naive accuracy achieved by feature extraction.”
Large Scale Incremental Learning (2019, methodology): “[9] presented a method for domain transfer learning.”
Learning Without Memorizing (2018, methodology): “TI methods LwF [12] IMM [10], EWC [9], MAS [1], [2], [8]”