Encoder Based Lifelong Learning

By Amal Rannen Triki, Rahaf Aljundi, Matthew B. Blaschko, Tinne Tuytelaars. 2017

348 citations recorded by the source corpus.

This paper introduces a new lifelong learning solution where a single model is trained for a sequence of tasks. The main challenge that vision systems face in this context is catastrophic forgetting: as they tend to adapt to the most recently seen task, they lose performance on the tasks that were learned previously. Our method aims at preserving the knowledge of the previous tasks while learning a new one by using autoencoders. For each task, an under-complete autoencoder is learned, capturing the features that are crucial for its achievement. When a new task is presented to the system, we prevent the reconstructions of the features with these autoencoders from changing, which has the effect of preserving the information on which the previous tasks are mainly relying. At the same time, the features are given space to adjust to the most recent environment as only their projection into a low dimension submanifold is controlled. The proposed system is evaluated on image classification tasks and shows a reduction of forgetting over the state-ofthe-art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “LwM [119] exploits the attention maps of new training samples for KD. EBLL [121] learns task-speciﬁc autoencoders and prevents changes in feature reconstruction.”
Measuring Catastrophic Forgetting in Neural Networks (2017, background): “While catastrophic forgetting occurs in unsupervised frameworks (Draelos et al. 2016; Goodrich and Arel 2014; Triki et al. 2017), we focus on supervised classiﬁcation.”
Gradient Episodic Memory for Continual Learning (2017, background): “Other approaches are instead based on “episodic” memory [Jung et al., 2016, Li and Hoiem, 2016, Rannen Triki et al., 2017, Rebufﬁ et al., 2017], where examples from previous tasks are stored and replayed to maintain predictions invariant by means of distillation [Hinton et al., 2015].”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, neutral): “Regularization-based methods can be divided into two classes: one is to distill knowledge from the previous model which is trained on the previous tasks [14, 15, 23, 36, 52, 59]; another is to explicitly use a regularizer to penalize the update of important parameters of previous tasks, preventing t”
Online Continual Learning with Contrastive Vision Transformer (2022, neutral): “Rehearsal-based methods [8,37,81,60,15,22,49,79,68,52] alleviate catastrophic forgetting by replaying a subset of data of past tasks stored in limited buffer.”
Continual Learning through Retrieval and Imagination (2022, background): “(2) trying to consolidate the knowledge in the original network by replaying a subset of past examples stored in a memory buffer (Buzzega et al. 2020; Hou et al. 2018; Rannen et al. 2017; Rebuffi et al. 2017; Buzzega et al. 2021).”
The Challenges of Continuous Self-Supervised Learning (2022, background): “In both cases, the model is expected to learn distinct tasks presented sequentially, without forgetting previous ones [32, 38, 64, 76, 85].”
Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector (2021, neutral): “The representative works include LwF [25], EBLL [32], GD-WILD [23], etc. Rehearsal methods : This line of works preserve existing information by replaying data from previous tasks.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[51] pointed out that LwF would not result in good performance if the data distributions between different tasks are quite diverse.”
Continual Learning for Neural Machine Translation (2021, methodology): “Following previous works on lifelong learning (Aljundi et al., 2017; Triki et al., 2017; Aljundi et al., 2018; Hou et al., 2019; Wu et al., 2019), we use a memory with ﬁxed capacity to reserve the training examples sampled from old data.”
Continual Learning for Named Entity Recognition (2021, methodology): “Previous work mostly focused on Computer Vision (Shmelkov, Schmid, and Alahari 2017; Li and Hoiem 2018; Rannen et al. 2017) by using Knowledge Distillation (Hin-ton, Vinyals, and Dean 2015) as the base framework.”
Continual Learning for Text Classification with Information Disentanglement Based Regularization (2021, methodology): “…models (Shin et al., 2017; Sun et al., 2019); (ii) regularization-based method, which constrains model’s output (Li and Hoiem, 2018), hidden space (Rannen et al., 2017), or parameters (Lopez-Paz and Ranzato, 2017; Zenke et al., 2017; Aljundi et al., 2018) from changing too much to retain…”
Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning (2021, methodology): “Other regularization methods imposed knowledge distillation [23] techniques to penalize the feature drift on previous tasks [35, 57, 45].”
Online Continual Learning in Image Classification: An Empirical Survey (2021, methodology): “KD has been widely adopted in CL methods [41, 12, 42, 43], and it is often considered as one of the regularization techniques.”
Do not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting (2021, methodology): “Other data-generation based methods [23, 33, 38, 39, 42] use generative adversarial network [9] and by replying the total data, new model is trained.”
Online Class-Incremental Continual Learning with Adversarial Shapley Value (2020, methodology): “Other regularization-based meth-ods imposed knowledge distillation techniques to penalize the feature drift on previous tasks (Li and Hoiem 2016; Wu et al. 2019; Rannen et al. 2017).”