Continual Learning in Deep Networks: an Analysis of the Last Layer

By Timothée Lesort, Thomas George, Irina Rish. 2021

24 citations recorded by the source corpus.

We study how different output layer parameterizations of a deep neural network affects learning and forgetting in continual learning settings. The following three effects can cause catastrophic forgetting in the output layer: (1) weights modifications, (2) interference, and (3) projection drift. In this paper, our goal is to provide more insights into how changing the output layer parameterization may address (1) and (2). Some potential solutions to those issues are proposed and evaluated here in several continual learning scenarios. We show that the best-performing type of output layer depends on the data distribution drifts and/or the amount of data available. In particular, in some cases where a standard linear layer would fail, changing parameterization is sufficient to achieve a significantly better performance, without introducing any continual-learning algorithm but instead by using standard SGD to train a model. Our analysis and results shed light on the dynamics of the output layer in continual learning scenarios and suggest a way of selecting the best type of output layer for a given scenario.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Continual Learning via Local Module Composition (2021, background): “The rapid growth of continual learning has lead researchers to work on empirical studies [20, 58, 56], surveys [32, 42, 55, 66, 67] as well as CL-specific software [72, 22, 59].”
Continual Learning with Foundation Models: An Empirical Study of Latent Replay (2022, methodology): “…the CL stream creation strategy used in this study can also be used for domain-incremental experimentation, for example, by creating a sequence of tasks with the same raw-dataset encoded with various encoders to simulate a representation drift (Caccia and Pineau, 2021; Lesort et al., 2021b).”
Efficient Continual Learning Ensembles in Neural Network Subspaces (2022, background): “One fundamental goal in continual learning is to learn from the new incoming tasks while retaining knowledge from the past and avoiding interference that can lead to poor performance (Lesort et al., 2021).”
Scaling the Number of Tasks in Continual Learning (2022, background): “In classical continual learning, forgetting still happens because the last layer is determinant for prediction and more sensitive to forgetting Wu et al. [2019], Hou et al. [2019], Zhao et al. [2020], Ramasesh et al. [2021], Bell and Lawrence [2021], Lesort et al. [2021a].”
Pretrained Language Model in Continual Learning: A Comparative Study (2022, background): “The rapid growth of continual learning has lead researchers to work on empirical studies (De Lange et al., 2019; Lesort et al., 2021b), surveys (Hadsell et al., 2020; Khetarpal et al., 2020; Lesort et al., 2021a; Mundt et al., 2020; 2021) as well as CL-speciﬁc software (Normandin et al., 2021;…”