Learning without Forgetting

By Zhizhong Li, Derek Hoiem. 2016

5,765 citations recorded by the source corpus.

When building a unified vision system or gradually adding new apabilities to a system, the usual assumption is that training data for all tasks is always available. However, as the number of tasks grows, storing and retraining on such data becomes infeasible. A new problem arises where we add new capabilities to a Convolutional Neural Network (CNN), but the training data for its existing capabilities are unavailable. We propose our Learning without Forgetting method, which uses only new task data to train the network while preserving the original capabilities. Our method performs favorably compared to commonly used feature extraction and fine-tuning adaption techniques and performs similarly to multitask learning that uses original task data we assume unavailable. A more surprising observation is that Learning without Forgetting may be able to replace fine-tuning with similar old and new task datasets for improved new task performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, neutral): “In fact, multi-task learning plus self training is similar to the Learning without Forgetting algorithm (Li and Hoiem, 2016), which trains the model to keep its predictions on an old task unchanged when learning a new task.”
What makes ImageNet good for transfer learning? (2016, methodology): “revent over-ﬁtting and what layers should be used for transfer learning was studied by [2,44]. A thorough investigation of good architectural choices for transfer learning was conducted by [3], while [26] propose an approach to ﬁne-tuning for new tasks without ”forgetting” the old ones. In contrast ”
Growing a Brain: Fine-Tuning by Increasing Model Capacity (2017, methodology): “Importance of reconciling the learning pace of new and old units: The previous work showed that network expansion did not introduce additional beneﬁts [22].”
The Curse of Recursion: Training on Generated Data Makes Models Forget (2023, methodology): “A typical way of preventing it is to use regularisations (Memory Aware Synpass [Aljundi et al., 2018]) or just rely on data ( e.g. Learning without Forgetting [Li and Hoiem, 2017]).”
Toward Understanding Catastrophic Forgetting in Continual Learning (2019, background): “In recent years, interest in continual learning has risen [1, 21, 26, 27, 29, 39, 40, 42, 51], especially from the deep learning research community, due to its potential to reduce training time and training set sizes (e.g., by continuously adapting from previous models), both of which are critical…”
Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines (2018, methodology): “s management is similar to iCaRL [11], except that we randomly pick the images for staying in the buffer. 4 For comparison, we pick several popular methods (EWC[3], online EWC[23], SI[9], MAS[24], LwF[25]) and state-of-the-art rehearsal-based methods (DGR[8], RtF[14]) with generative model. The memo”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, methodology): “We were worried, however, that the these representations would drift and so we added an auxiliary loss to the model inspired by [48]: the 2048-dimensional representation of each object (without class embeddings) had to be predictive of that object’s label (via a linear projection to the label space ”
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks (2016, background): “In the field of computer vision, some transfer and multi-task learning approaches have also been proposed (Li and Hoiem, 2016; Misra et al., 2016).”
Born Again Neural Networks (2018, background): “A number of recent papers (Furlanello et al., 2016; Li & Hoiem, 2016; Shin et al., 2017) employ KD for the purpose of minimizing forgetting in continual learning.”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, neutral): “Remedies to this problem can include using the original model outputs as supervision signals [7, 36, 52], or, as advocated in [28], considering the sensitivity of other tasks when updating shared parameters.”
How To Backdoor Federated Learning (2018, background): “Preventing the backdoor from being forgotten as the model is updated by benign participants is similar to the catastrophic forgetting problem in multi-task learning [22, 33, 39].”
AutoML: A Survey of the State-of-the-Art (2019, methodology): “Li and Hoiem [147] propose Learning without Forgetting (LwF) method, which trains the model only using new data, while preserving the original capabilities. iCaRL [148] is a new training strategy based on LwF, which only uses a small part of old data to pretrained the information and gradually incre”
Continual Learning with Deep Generative Replay (2017, methodology): “A recently proposed method called Learning without Forgetting (LwF) [21] addresses the problem of sequential learning in image classification tasks while minimizing alteration on shared network parameters.”
Experience Replay for Continual Learning (2018, methodology): “atent dimensions / states that protect information. Outside of RL, Zenke et al. [32] develop a method similar to EWC that maintains estimates of the importance of weights for past tasks, Li and Hoiem [15] leverage a mixture of task-speciﬁc and shared parameters, and Milan et al. [19] develop a rigor”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Asapioneerwork,LwF[118]andLwF.”
Variational Continual Learning (2017, background): “A more elegant and more flexible approach maintains a single model and uses a single type of regularized training that prevents drastic changes in the parameters which have a large influence on prediction, but allows other parameters to change more freely (Li and Hoiem, 2016; Kirkpatrick et al., 201”
Knowledge Flow: Improve Upon Your Teachers (2019, methodology): “…transfer have been considered using a variety of techniques, for instance, ﬁne-tuning, progressive neural nets (Rusu et al., 2016b), PathNet (Fernando et al., 2017), ‘Growing a Brain’ (Wang et al., 2017), actor-mimic (Parisotto et al., 2016), learning without forgetting (Li & Hoiem, 2016).”
Variational Information Distillation for Knowledge Transfer (2019, methodology): “This was later extended for knowledge transfer between the teacher and the student networks designed for different tasks by Li and Hoiem [16], through adding an additional linear layer on top of the penultimate layer S in the student network to matching with logit layer T (logit) in the teacher netw”