Catastrophic Forgetting and the Pseudorehearsal Solution in Hopfield-type Networks

By Anthony V. Robins, Simon McCallum. 1998

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Structural Synaptic Plasticity Has High Memory Capacity and Can Explain Graded Amnesia, Catastrophic Forgetting, and the Spacing Effect (2014, background): “Alternative high level mechanisms for preventing CF involve pseudo-rehearsal using self-generated training stimuli from previously learned memories [92].”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, positive): “This mechanism is the basis of replay in computational models of attractor networks (Crick & Mitchison, 1983; Robins & McCallum, 1998, 1999).”
Functional Regularisation for Continual Learning using Gaussian Processes (2019, background): “This idea has been explored before (e.g. Robins & McCallum, 1998) although recent progress in generative modelling continues to make the idea of pseudorehearsals an attractive and simple alternative.”
Label Mapping Neural Networks with Response Consolidation for Class Incremental Learning (2019, methodology): “A general and straightforward solution to the catastrophic forgetting problem is Rehearsal [11]: the neural network is able to retrain by the old classes while learning new classes.”
Critical periods and catastrophic interference effects in the development of self-organizing feature maps. (2008, background): “…typically qualitative in nature (e.g. catastrophic interference effects: French & Chater, 2002; McCloskey & Cohen, 1989; Ratcliff, 1990; Robins, 1995; Robins & McCallum, 1998; Sharkey & Sharkey, 1995; critical period effects: McClelland et al., 1999; Oliver et al., 2000; O’Reilly & Johnson, 1994).”
Controlling Recurrent Neural Networks by Conceptors (2014, background): “Although a number of proposals have been made which partially alleviate the problem in special circumstances ([32, 42, 73], skeptical overview: [77]), catastrophic forgetting was still considered a main challenge for neural learning theory in an expert’s hearing solicited by the NSF in 2007 ([21]).”
Structural plasticity, cortical memory, and the spacing effect (2009, unknown): “In this work, I present additional simulations and analyses suggesting that networks employing structural plasticity suffer to a much lesser degree from catastrophic forgetting than models without structural plasticity [3] if the number of consolidated synapses remains sufficiently low.”
Sequential learning in neural networks: A review and a discussion of pseudorehearsal based methods (2004, unknown): “Figure 4 summarises results from simulations in [58].”
Catastrophic forgetting in simple networks: an analysis of the pseudorehearsal solution. (1999, background): “Similar issues have been explored in Hopfield networks [15, 1, 20]§.”
Neural networks with a self-refreshing memory: Knowledge transfer in sequential learning tasks without catastrophic forgetting (2000, background): “It must be noticed that pseudorehearsal was explored in the framework of Hop eld-type nets (Robins and McCallum 1998), where extra (or ‘spurious’) attractors created in state space during learning are exploited for preserving a previously learned population.”
The consolidation of learning during sleep: comparing the pseudorehearsal and unlearning accounts (1999, unknown): “The two methods explored are the " pseudorehearsal " method (Robins, 1995; Robins & McCallum, 1998) and the " unlearning " method (Hopfield, Feinstein & Palmer, 1983; Crick & Mitchison, 1983).”