Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem

By M. McCloskey, N. J. Cohen. 1989

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Learning and Evaluating General Linguistic Intelligence (2019, background): “…words into representations of phrases, sentences, and documents), and previously acquired linguistic knowledge to avoid catastrophic forgetting (McCloskey & Cohen, 1989; French, 1999), and (iii) adapt to new linguistic tasks in new environments with little experience (i.e., robustness to…”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “Much effort has gone into mitigating catastrophic forgetting (McCloskey and Cohen, 1989; Ratcliff, 1990; Kemker et al., 2017) by penalizing the norm of parameters when training on a new task (Kirkpatrick et al.”
Simple, Scalable Adaptation for Neural Machine Translation (2019, methodology): “tion task, our approach circumvents catastrophic interference (McCloskey and Cohen, 1989) with the original model parameters, and allows us to simultaneously adapt a single model to multiple domains and languages, while retaining the quality on the source languages and domains.”
Parameter-Efficient Transfer Learning for NLP (2019, background): “However, when re-trained, deep networks tend to forget how to perform previous tasks; a challenge termed catastrophic forgetting (McCloskey & Cohen, 1989; French, 1999).”
Detoxifying Language Models Risks Marginalizing Minority Voices (2021, methodology): “This technique aims to erase an LM’s knowledge of toxicity via catastrophic forgetting (McCloskey and Cohen, 1989).”
Toward Understanding Catastrophic Forgetting in Continual Learning (2019, background): “However, continual learning by deep models has proven to be very challenging due to catastrophic forgetting , a long known problem of training deep neural networks [4, 5, 14, 15, 28, 31, 34].”
Overcoming catastrophic forgetting in neural networks (2016, background): “This phenomenon, termed catastrophic forgetting [French, 1999, McCloskey and Cohen, 1989, McClelland et al., 1995, Ratcliff, 1990], occurs speciﬁcally when the network is trained sequentially on multiple tasks because the weights in the network that are important for task A are changed to meet the…”
The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, background): “Likewise, it has been known for three decades that they are vulnerable to a problem known as catastrophic interference in which earlier associations are overwritten by later associations (McCloskey & Cohen, 1989), making them highly sensitive to the sequence in which items are presented.”
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, background): “Models are also prone to catastrophic forgetting, where the LM loses its ability to do things that it was able to do before ine-tuning [84].”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “One recent line of work [26, 68, 78, 114, 136, 144] proposes to freeze the pretrained LM weights to prevent catastrophic forgetting [71].”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, methodology): “However, as we subsequently ﬁne-tune the model weights on new tasks, the problem of catastrophic forgetting (McCloskey and Cohen, 1989; French, 1999) can arise, which results in loss of knowledge already learned from all previous tasks.”
Meta-learning with differentiable closed-form solvers (2018, methodology): “In order to alleviate the drawback of catastrophic forgetting typical of deep neural networks [38], several recent methods [19–22] make use of memory-augmented models, which can first retain and then access important and previously unseen information associated with newly encountered tasks.”
Fast Model Editing at Scale (2021, neutral): “, 1989; Ratcliff, 1990; Kirkpatrick et al., 2017).”
Neural Programmer-Interpreters (2015, background): “The learning of new tasks may require that the network weights change substantially, so care must be taken to avoid catastrophic forgetting (Mccloskey & Cohen, 1989; OReilly et al., 2014).”
Learning without Forgetting (2016, background): “Ideally, the new tasks could be learned while sharing parameters from old ones, without suffering from Catastrophic Forgetting [1], [2] (degrading performance on old tasks) or having access to the old training data.”
Continual Lifelong Learning with Neural Networks: A Review (2018, background): “ning is that they are prone to catastrophic forgetting or catastrophic interference, i.e., training a model with new information interferes with previously learned knowledge (McClelland et al., 1995; McCloskey and Cohen, 1989). This phenomenon typically leads to an abrupt performance decrease or, in”
Continual Learning with Deep Generative Replay (2017, background): “The term catastrophic forgetting or catastrophic interference was first introduced by McCloskey and Cohen in 1980’s [22].”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “A major challenge is known as catastrophic forgetting [6], [7], where adaptation to a new distribution generally results in a largely reduced ability to capture the old ones.”