Catastrophic forgetting in neural networks: the role of rehearsal mechanisms

By Anthony V. Robins. 1993

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Meta & Continual Learning

Citations in the corpus

Continual Lifelong Learning with Neural Networks: A Review (2018, background): “, & Robins, A. (2005).Memory retention andweight plasticity in ANN”
Continual Learning with Deep Generative Replay (2017, background): “Several works illustrate empirical consequences in sequential learning settings [7, 29], and provide a few primitive solutions [16, 30] such as replaying all previous data.”
The Ideal Continual Learner: An Agent That Never Forgets (2023, neutral): “, 2021; Heckel, 2022), memory-based methods (Robins, 1993; Shin et al., 2017; Rebuffi et al., 2017; Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019; Aljundi et al., 2019b; Buzzega et al., 2020; Verwimp et al., 2021; Bang et al., 2021; Kim et al., 2022; Zhang et al., 2022; Wang et al., 2022a; Saha &”
Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System (2022, methodology): “CLS has been used as a source of inspiration for dual memory learning systems in earlier works (French, 1999; Robins, 1993) but they have not been shown to scale to current computer vision tasks (Parisi et al., 2019).”
DER: Dynamically Expandable Representation for Class Incremental Learning (2021, background): “Among them, perhaps the most effective strategy is to keep a memory buffer that stores part of observed data for the rehearsal [28, 29] in future.”
Continual Learning with Adaptive Weights (CLAW) (2019, methodology): “(c) The memory-based approach relies on episodic memory to store data (or pseudo-data) from previous tasks (Ratcliff, 1990; Robins, 1993; 1995; Thrun, 1996; Schmidhuber, 2013; Hattori, 2014; Mocanu et al., 2016; Rebuffi et al., 2017; Kamra et al., 2017; Shin et al., 2017; Rolnick et al., 2018; van d”
Generative Feature Replay with Orthogonal Weight Modification for Continual Learning (2020, unknown): “Replay (also called rehearsal) strategy is initially proposed to relearn a subset of previously learned data when learning the current task [Robins, 1993].”
The Primacy Bias in Deep Reinforcement Learning (2022, result): “Forgetting mechanisms In contrast to the well-known phenomenon of catastrophic forgetting (French, 1999), several works have noted the opposite effect of catastrophic memorization (Robins, 1993; Sharkey & Sharkey, 1995) similar to the primacy bias.”
Catastrophic Forgetting, Rehearsal and Pseudorehearsal (1995, methodology): “We have duplicated Ratcliff's results using exactly the same architecture and training methods that he did (see Robins (1993)).”
CILEA-NET: Curriculum-Based Incremental Learning Framework for Remote Sensing Image Classification (2021, background): “Early attempts in alleviating the effect of catastrophic forgetting consisted of storing old data and replaying them repeatedly in an interleaved fashion along with the new samples [5].”
Mitigation of catastrophic forgetting in recurrent neural networks using a Fixed Expansion Layer (2013, methodology): “Standard rehearsal methods set samples of data aside during training and later reuse these samples in order to prevent loss of information; sweep rehearsal [18] and the rehearsal buffer model [17] are two examples of these.”
Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping (2021, methodology): “Since Robins [43] showed the promise of memory replay methods in dealing with CF and the prevalence of cognitive/neuro science inspired theories regarding memory, it is of no surprise that rehearsal,memory buffer or generative replay, etc.”
Learning in situ: a randomized experiment in video streaming (2019, methodology): “16], the problem that neural networks trained with new data may lose the parameters to perform well on the old data. Therefore, we apply a technique in continual learning known as “experience replay” [23,24] to mitigate catastrophic forgetting. Our experience-replay strategy is relatively simple: we”
Deep Class-Incremental Learning: A Survey (2023, background): “‘Replay’ is important in human cognition system [119] — a student facing final exams shall go over the textbooks to recall former memory and knowledge.”
Continual Learning for Affective Robotics: Why, What and How? (2020, background): “2) Rehearsal-based Approaches: To mitigate forgetting in incremental learning, a straightforward approach can be to physically store the encountered data in memory and regularly replay it (known as rehearsal), interleaved with new samples [36].”
Overcoming Long-Term Catastrophic Forgetting Through Adversarial Neural Pruning and Synaptic Consolidation (2019, background): “In works [7] [8] [9] [10] based on the rehearsal strategy, previous memories are reinforced by replaying experiences.”
TAG: Task-based Accumulated Gradients for Lifelong learning (2021, background): “This leads to a significant drop in performance on the previous tasks - also known as catastrophic forgetting [38, 47].”
Task Agnostic Representation Consolidation: a Self-supervised based Continual Learning Approach (2022, background): “Early works attempted to mitigate the effect of catastrophic forgetting by replaying a subset of training data from the previous tasks stored in the replay buffer alongside samples from the new task (Robins, 1993).”