Using Semi-Distributed Representations to Overcome Catastrophic Forgetting in Connectionist Networks

By R. M. French. 1991

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Meta & Continual Learning

Citations in the corpus

Overcoming catastrophic forgetting with hard attention to the task (2018, background): “The other popular strategy to overcome catastrophic forgetting is to reduce representational overlap (French, 1991).”
Class-Incremental Continual Learning Into the eXtended DER-Verse (2022, background): “distinct parts of themodel at distinct times [8], [9], or by revisiting past data [10], [11], [12], CL approaches let current training be influenced by previously learned information to preserve it.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Additionally, when gradient based learning and function approximation are applied to multiple diverse tasks simultaneously, this is known to often result in interfering gradients (Caruana, 1997; French, 1991) across tasks.”
Learning to Continually Learn (2020, background): “Another approach to mitigate CF is to directly incentivize the creation of maximally sparse or disjoint representations [11, 26], with the goal of minimizing interference between activations.”
Meta-Learning Representations for Continual Learning (2019, background): “Second, they suffer from catastrophic interference when learning online from a correlated stream of data (French, 1991).”
Continual Learning via Neural Pruning (2019, methodology): “Our method also bears some resemblance to [3], a forbear of clustering based methods put forward in the 90s.”
Learning to Learn without Forgetting By Maximizing Transfer and Minimizing Interference (2018, background): “, 2018), the link to weight sharing (French, 1991; Ajemian et al., 2013), and the idea of influencing gradient alignment for continual learning before (Lopez-Paz & Ranzato, 2017).”
The Utility of Sparse Representations for Control in Reinforcement Learning (2018, background): “Early work on catastrophic interference investigated some simple heuristics for encouraging sparsity, such as node sharpening (French 1991).”
Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. (1995, background): “In fact, several investigators have demonstrated ways of reducing the magnitude of interference in tasks like those studied by McCloskey and Cohen (1989; French, 1991, 1992; Hetherington & Seidenberg, 1989; Kortge, 1993; McRae & Hetherington, 1993; Sloman & Rumelhart, 1992).”
Unsupervised neuron selection for mitigating catastrophic forgetting in neural networks (2014, background): “An older approach, known as activation sharpening, attempted to achieve this [5].”
Automatic Recall Machines: Internal Replay, Continual Learning and the Brain (2020, background): “Behaviour generalises to patterns not present in training, which is not true of non-distributed representations such as lookup tables [French, 1991].”
Neuron clustering for mitigating catastrophic forgetting in feedforward neural networks (2014, methodology): “A technique known as activation sharpening [6] directly accomplished this very goal.”
Does an LSTM forget more than a CNN? An empirical study of catastrophic forgetting in NLP (2019, background): “Previous studies found that using dropout (Goodfellow et al., 2014) and sharp activation functions (French, 1991) help reduce catastrophic forgetting.”
Dynamically constraining connectionist networks to produce distributed, orthogonal representations to reduce catastrophic interference (2019, methodology): “French (1991), Murre (1992) and Lewandowsky & Goebel (1991) have shown that this technique was effective in reducing activation overlap among hidden-layer representations which, in turn, decreased catastrophic forgetting.”
SpaceNet: Make Free Space For Continual Learning (2020, result): “It is worth highlighting that our findings from this research are aligned with the early work by French [8].”