Catastrophic Forgetting, Rehearsal and Pseudorehearsal

By Anthony V. Robins. 1995

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Meta & Continual Learning

Citations in the corpus

The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “…learned tasks during parameter updates [Hashimoto et al., 2016], incrementally matching modes [Lee et al., 2017], rehearsing on old tasks [Robins, 1995], using adaptive memory buffers [Gepperth and Karaoguz, 2016], finding task-specific paths through networks [Fernando et al., 2017], and…”
Toward Understanding Catastrophic Forgetting in Continual Learning (2019, background): “process analogous to human learning. However, continual learning by deep models has proven to be very challenging due to catastrophic forgetting, a long known problem of training deep neural networks [4, 5, 14, 15, 28, 31, 34]. Catastrophic forgetting refers to the tendency of a model to forget all ”
Continual Lifelong Learning with Neural Networks: A Review (2018, background): “French (1997) developed a pseudo-recurrent dual-memory framework, one for early processing and the other for long-termstorage, that usedpseudo-rehearsal (Robins, 1995) to transfer memories between memory centres.”
Continual Learning with Deep Generative Replay (2017, methodology): “Called a pseudorehearsal technique, this method is claimed to maintain old input-output patterns without accessing real data [31].”
iCaRL: Incremental Classifier and Representation Learning (2016, background): “…standard classification loss, which encourages improvements of the feature representation that allow classifying the newly observed classes well, it also contains the distillation loss, which ensures that the discriminative information learned previously is not lost during the new learning step.”
Lifelong Learning of Spatiotemporal Representations With Dual-Memory Recurrent Self-Organization (2018, methodology): “replay (Robins, 1995) and has the advantage of fewer memory requirements with respect to explicitly storing training samples.”
Overcoming catastrophic forgetting with hard attention to the task (2018, background): “Storing a certain amount of previous information and us-ing it to retrain the model was among the earliest attempts to overcome catastrophic forgetting; a strategy named “re-hearsal” (Robins, 1995).”
FearNet: Brain-Inspired Model for Incremental Learning (2017, background): “FearNet mitigates catastrophic forgetting by consolidating recent memories into long-term storage using pseudorehearsal (Robins, 1995).”
Deep Generative Dual Memory Network for Continual Learning (2017, background): “for sequential learning is experience replay. McClelland et al. (1995); O’Neill et al. (2010) have emphasized the importance of replayed data patterns in the human brain during sleep and waking rest. Robins (1995; 2004) proposed several replay techniques (a.k.a. pseudopattern rehearsal) to achieve r”
Measuring Catastrophic Forgetting in Neural Networks (2017, methodology): “Rehearsal methods try to mitigate catastrophic forgetting by mixing data from earlier batches with the current batch to be learned (Robins 1995).”
Born to Learn: the Inspiration, Progress, and Future of Evolved Plastic Artificial Neural Networks (2017, background): “Plasticity in neural networks, when continuously active, was also observed to cause catastrophic forgetting (Robins, 1995).”
The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects (2013, background): “There are a number of ways of overcoming this difﬁculty, for instance by manipulating the orthogonality or the sparseness of the input-output patterns (French, 1992; Robins, 1995).”
Progress & Compress: A scalable framework for continual learning (2018, background): “to keep weights of columns tied. Another category of work is based on the idea of episodic memory, where examples from prior tasks are stored to effectively recall experience encountered in the past (Robins, 1995). Examples of this method are (Rebufﬁ et al.,2016; Thrun,1996). A similar approach is p”
Task-Aware Information Routing from Common Representation Space in Lifelong Learning (2023, background): “Similar to experience rehearsal in the brain, early works attempted to address catastrophic forgetting through Experience-Replay (ER; Ratcliff (1990); Robins (1995)) by explicitly storing and replaying previous task samples alongside current task samples.”
Few-Shot Continual Active Learning by a Robot (2022, methodology): “To mitigate catastrophic forgetting, we use pseudo-rehearsal [20] using the samples generated from the Gaussian distributions of the old classes, thus removing the need to store raw data for the classes.”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “For instance, memory-based methods alleviate forgetting by either replaying a small set of examples from past tasks saved in a memory [57, 55, 46, 7, 23, 28, 9] or replaying synthesized data from previous tasks using generative models [60, 63].”
Transfer without Forgetting (2022, background): “– Experience Replay (ER) [49,53] is the first embodiment of a rehearsal strategy that features a small memory buffer containing an i.”
Information-theoretic Online Memory Selection for Continual Learning (2022, neutral): “Continual learning (Robins, 1995; Goodfellow et al., 2013; Kirkpatrick et al., 2017) aims at training models through a non-stationary data stream without catastrophic forgetting of past experiences.”