Catastrophic forgetting in connectionist networks.

By R. M. French. 1999

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Learning and Evaluating General Linguistic Intelligence (2019, background): “…into representations of phrases, sentences, and documents), and previously acquired linguistic knowledge to avoid catastrophic forgetting (McCloskey & Cohen, 1989; French, 1999), and (iii) adapt to new linguistic tasks in new environments with little experience (i.e., robustness to domain shifts).”
Parameter-Efficient Transfer Learning for NLP (2019, background): “However, when re-trained, deep networks tend to forget how to perform previous tasks; a challenge termed catastrophic forgetting (McCloskey & Cohen, 1989; French, 1999).”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “, 2010) and avoiding catastrophic forgetting (French, 1999; Kirkpatrick et al., 2017) have a particular flavor in the domain of conversation: the general task is always to talk to people about any subject, but the set of people, the topics of interest, the facts that have happened may all change.”
Toward Understanding Catastrophic Forgetting in Continual Learning (2019, background): “However, continual learning by deep models has proven to be very challenging due to catastrophic forgetting , a long known problem of training deep neural networks [4, 5, 14, 15, 28, 31, 34].”
ProoFVer: Natural Logic Theorem Proving for Fact Verification (2021, background): “However, this comes at a performance degradation of more than 9 % on the original FEVER development data for both KernelGAT and CorefBERT, due to catastrophic forgetting (French, 1999).”
Generative Adversarial Networks (2021, neutral): “Neural networks have been shown to forget previous learned tasks [17, 35], and catastrophic forgetting was previously considered as a major cause for GAN training instability.”
Self-Supervised GANs via Auxiliary Rotation Loss (2018, background): “In non-stationary online environments, neural networks forget previous tasks [11, 12, 13].”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, methodology): “However, as we subsequently ﬁne-tune the model weights on new tasks, the problem of catastrophic forgetting (McCloskey and Cohen, 1989; French, 1999) can arise, which results in loss of knowledge already learned from all previous tasks.”
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation (2016, neutral): “language pair B) is incorporated (French, 1999).”
Continual Lifelong Learning with Neural Networks: A Review (2018, background): “This review complements previous surveys on catastrophic forgetting in connectionist models (French, 1999; Goodfellow, Mirza, Xiao, Courville, & Bengio, 2013; Soltoggio, Stanley, & Risi, 2017) that do not critically compare recent experimental work (e.”
Continual Learning with Deep Generative Replay (2017, background): “Several works illustrate empirical consequences in sequential learning settings [7, 29], and provide a few primitive solutions [16, 30] such as replaying all previous data.”
Experience Replay for Continual Learning (2018, background): “The problem of catastrophic forgetting in neural networks has long been recognized [6], and it is known that rehearsing past data can be a satisfactory antidote for some purposes [4, 18].”
Salvaging Federated Learning by Local Adaptation (2020, methodology): “To overcome the catastrophic forgetting (French, 1999) of task X while learning Y , we use elastic weight consolidation (Kirkpatrick et al., 2017) which selectively slows down learning on the weights important for X .”
iCaRL: Incremental Classifier and Representation Learning (2016, background): “…standard classification loss, which encourages improvements of the feature representation that allow classifying the newly observed classes well, it also contains the distillation loss, which ensures that the discriminative information learned previously is not lost during the new learning step.”
Continuous Learning in Single-Incremental-Task Scenarios (2018, background): “Unfortunately, when (deep or shallow) neural networks are trained only on new data, they experience a rapid overriding of their weights with a phenomenon known in literature as catastrophic forgetting [22][26][2].”
FearNet: Brain-Inspired Model for Incremental Learning (2017, methodology): “In French (1999), methods developed in the 1980s and 1990s are extensively discussed, and French argued that mitigating catastrophic forgetting would require having two separate memory centers: one for the long-term storage of older memories and another to quickly process new information as it…”