Catastrophic Interference is Eliminated in Pretrained Networks

By Ken McRae, A. Hetherington. 1993

Subfields: Model Architectures, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Meta & Continual Learning

Citations in the corpus

Deep Generative Dual Memory Network for Continual Learning (2017, background): “…works attempted to remedy the issue by reducing representational overlap between input representations via activation sharpening algorithms (Kortge, 1990), orthogonal recoding of inputs (Lewandowsky, 1991) or orthogonal activations at all hidden layers (McRae & Hetherington, 1993; French, 1994).”
The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects (2013, background): “…it is possible to reduce the problem of catastrophic forgetting by reducing the overlap among the internal representations stored in the neural network, for example using larger systems, or for example sparse or interleaved learning (Hetherington and Seidenberg, 1989; McRae and Hetherington, 1993).”
RMM: Reinforced Memory Management for Class-Incremental Learning (2023, background): “However, deep-learning-based AI systems, when continuously updated using new data and limited historical data, often suffer from catastrophic forgetting, as the updates can override knowledge acquired from previous data [33, 34, 39].”
Adaptive Aggregation Networks for Class-Incremental Learning (2020, background): “This is due to two facts: (i) the updates can override the knowledge acquired from the previous data [19, 27, 28, 33, 40], and (ii) the model can not replay the entire previous data to regain the old knowledge.”
Mnemonics Training: Multi-Class Incremental Learning Without Forgetting (2020, background): “In contrast, current machine learning systems, when continuously updated using novel incoming data, suffer from catastrophic forgetting (or catastrophic interference), as the updates can override knowledge acquired from previous data [19, 20, 23, 27, 12].”
The consolidation of learning during sleep: comparing the pseudorehearsal and unlearning accounts (1999, unknown): “Catastrophic forgetting may also be prevented if it is possible to pre-train the network on a population which is " relevant " to the base population and new items (simulating prior knowledge of a task domain) as explored by Sharkey and Sharkey (1995), McRae and Hetherington (1993).”
Catastrophic interference in connectionist networks (2003, background): “McRae and Hetherington (1993) eliminated the problem by pre-training the network on a random sample of patterns drawn from the domain.”
The Division of Labor Between the Neocortex and Hippocampus (2010, background): “A number of researchers showed that this interference can be reduced by introducing various factors that result in less pattern overlap (e.g., Kortge, 1993; French, 1992; Sloman & Rumelhart, 1992; McRae & Hetherington, 1993).”
Meta-Aggregating Networks for Class-Incremental Learning (2020, unknown): “This is due to two facts: (i) the updates can override the knowledge acquired from the previous data (McCloskey & Cohen, 1989; McRae & Hetherington, 1993; Ratcliff, 1990; Shin et al., 2017; Kemker et al., 2018); and (ii) the model can not replay the entire previous data to regain the old knowledge.”
Dynamically constraining connectionist networks to produce distributed, orthogonal representations to reduce catastrophic interference (2019, background): “A number of authors (Murre, 1992; McRae & Hetherington, 1993) also developed techniques that resulted in improved orthogonalization of representations at the hidden layer and they, too, observed similar decreases in catastrophic interference.”
Catastrophic Forgetting and the Pseudorehearsal Solution in Hopfield-type Networks (1998, methodology): “This problem has been identified in many forms in memory literature – it is commonly called the “catastrophic forgetting”, “catastrophic interference”, or “serial learning” problem (Hetherington and Seidenberg, 1989; McCloskey and Cohen, 1989; Lewandowsky, 1991; French, 1992; McRae and Hetherington,”
Sequential learning in neural networks: A review and a discussion of pseudorehearsal based methods (2004, unknown): “Examples of this approach include the novelty rule [34], activation sharpening [17], context biasing [18] and techniques described in [41,46].”
Consolidation in Neural Networks and in the Sleeping Brain (1996, methodology): “The novelty rule (Kortge 1990), activation sharpening (French 1992), techniques developed by Murre (1992) and McRae & Hetherington (1993), and context biasing (French 1994), all fall within this general framework.”
Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. (1995, background): “In fact, several investigators have demonstrated ways of reducing the magnitude of interference in tasks like those studied by McCloskey and Cohen (1989; French, 1991, 1992; Hetherington & Seidenberg, 1989; Kortge, 1993; McRae & Hetherington, 1993; Sloman & Rumelhart, 1992).”
Meta-learnt priors slow down catastrophic forgetting in neural networks (2019, background): “…SeqFOMAML on a more challenging scenario, and of exploring the effect of “good”, reusable features learnt by standard (non-sequential) meta-learning (Raghu et al., 2019; Srivatanapa, 2020), in mitigating catastrophic forgetting (McRae & Hetherington, 1993; Javed & White, 2019; French, 1999).”