Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics

By Vinay Venkatesh Ramasesh, Ethan Dyer, Maithra Raghu. 2020

216 citations recorded by the source corpus.

A central challenge in developing versatile machine learning systems is catastrophic forgetting: a model trained on tasks in sequence will suffer significant performance drops on earlier tasks. Despite the ubiquity of catastrophic forgetting, there is limited understanding of the underlying process and its causes. In this paper, we address this important knowledge gap, investigating how forgetting affects representations in neural network models. Through representational analysis techniques, we find that deeper layers are disproportionately the source of forgetting. Supporting this, a study of methods to mitigate forgetting illustrates that they act to stabilize deeper layers. These insights enable the development of an analytic argument and empirical picture relating the degree of forgetting to representational similarity between tasks. Consistent with this picture, we observe maximal forgetting occurs for task sequences with intermediate similarity. We perform empirical studies on the standard split CIFAR-10 setup and also introduce a novel CIFAR-100 based task approximating realistic input distribution shift.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Safety, Robustness & Interpretability

Citations in the corpus

A General Language Assistant as a Laboratory for Alignment (2021, background): “The fact that larger models are less subject to forgetting [RDR20] may be related to the fact that larger models do not incur signiﬁcant alignment taxes.”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “in function space, such as by using pre-training [167], [295], [352], wider network architectures [305], [351], [352] and self-supervised learning [57], [167], [281], [335].”
Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (2022, positive): “This result agrees with recent ﬁnding (Ramasesh et al., 2021) that deeper layers are more responsible for causing forgetting because their hidden representation deviates from the model trained on all tasks.”
Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions (2022, background): “Continual learning in practice The study of continual learning (or lifelong learning) dates back to the work of [34] and it receives a surge of research interest over recent years [14, 17, 10, 5, 13, 29, 32, 16, 27, 37].”
An Empirical Investigation of the Role of Pre-training in Lifelong Learning (2021, background): “It may be possible to manually cluster tasks based upon semantic closeness, rendering severe interference to make these benchmarks more challenging (Ramasesh et al., 2020).”
Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning (2021, background): “An empirical study of the cause of CF and the impact of task similarity on CF was done in [44].”
Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting (2021, neutral): “This is complementary to earlier studies on incremental learning of new classes and its catastrophic forgetting (Ramasesh et al., 2020; Lee et al., 2021), where each task includes different classes and does not allow the revisit.”
Kernel Continual Learning (2021, background): “Moreover, knowledge tends to be shared and transferred across tasks more in the lower layers than higher layers of deep neural networks (Ramasesh et al., 2021).”
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, methodology): “These works consistently observed a counter-intuitive role of the similarity between tasks A and B, with intermediate task similarity leading to worst forgetting (Ramasesh et al., 2020; Doan et al., 2020; Nguyen et al., 2019).”
Linear Mode Connectivity in Multitask and Continual Learning (2020, background): “Although, the phenomenon observed in (Ramasesh et al., 2020) is still realizable by a very tiny margin.”
A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix (2020, background): “, 2019], neural network representations level [Ramasesh et al., 2020] ), or proposed a quantitative metric [Farquhar and Gal, 2018, Kemker et al.”
Regularization Shortcomings for Continual Learning (2019, methodology): “As noted by [5,22] when tasks have similarities (e.”
Scaling the Number of Tasks in Continual Learning (2022, background): “In classical continual learning, forgetting still happens because the last layer is determinant for prediction and more sensitive to forgetting Wu et al. [2019], Hou et al. [2019], Zhao et al. [2020], Ramasesh et al. [2021], Bell and Lawrence [2021], Lesort et al. [2021a].”
A Closer Look at Rehearsal-Free Continual Learning * (2022, methodology): “We borrow the practice from [45] and look at the centered kernel alignment (CKA) similarity between feature representations over time for different layers in the model (higher is better).”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, background): “Another study empirically evaluated the effect of task semantics on catastrophic forgetting, finding that intermediate similarity leads to the highest amount of forgetting (Ramasesh et al., 2021).”
Continual Learning with Foundation Models: An Empirical Study of Latent Replay (2022, background): “Second, as discussed by Ramasesh et al. (2020), orthogonal representations lead to less forgetting due to less interference.”
Class Similarity Weighted Knowledge Distillation for Continual Semantic Segmentation (2022, methodology): “Following recent studies [26, 30], the centered kernel alignment (CKA) metric is used to measure the similarity of model representations.”
How catastrophic can catastrophic forgetting be in linear regression? (2022, background): “11) [56] Vinay Venkatesh Ramasesh, Ethan Dyer, and Maithra Raghu.”