Semi-distributed Representations and Catastrophic Forgetting in Connectionist Networks

By R. M. French. 1992

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Structural Synaptic Plasticity Has High Memory Capacity and Can Explain Graded Amnesia, Catastrophic Forgetting, and the Spacing Effect (2014, background): “Such approaches include ART-type architectures [90], emergent category-specific modularity [93], hard-wired modularity [94], and also ideas involving grandmother cells [95] or, in technical terms, look-up-tables [16].”
Online Continual Learning with Maximally Interfered Retrieval (2019, background): “While these problems have been studied in early research works [10, 11, 20, 21, 37], they are receiving increased attention since the revival of neural networks.”
Continual learning: A comparative study on how to defy forgetting in classification tasks (2019, background): “Several directions have been explored, such as reducing representation overlap [22], [34], [35], [36], [37], replaying samples or virtual samples from the past [10], [24] or introducing dual architectures [38], [39], [40].”
Selfless Sequential Learning (2018, methodology): “This has led to several methods with the goal of orthogonalizing the activations (French, 1992; 1994; Kruschke, 1992; 1993; Sloman & Rumelhart, 1992).”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, methodology): “For now, we will start with a typical categorization of existing deep continual works into the three categories of regularization, rehearsal and architectural approaches, in consistency ∙ LWF (LiandHoiem,2016) ∙ EBLL (Rannenetal.,2017) ∙ Pseudorehearsal (Robins,1995) ∙ Pseudo-recurrentnets (French,1”
Diffusion-based neuromodulation can eliminate catastrophic forgetting in simple neural networks (2017, background): “This isolation should help disentangle the parts of the ANN that encode for different aspects of problems [5].”
Connectionist modelling in psychology: A localist manifesto (2000, background): “French (1992) discusses two systems he describes as semidistributed.”
Continual learning in recurrent neural networks (2020, unknown): “Lastly, some regularization approaches rely on the use of non-overlapping and orthogonal representations to overcome catastrophic forgetting (French, 1992; 1994; 1970).”
Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. (1995, background): “In fact, several investigators have demonstrated ways of reducing the magnitude of interference in tasks like those studied by McCloskey and Cohen (1989; French, 1991, 1992; Hetherington & Seidenberg, 1989; Kortge, 1993; McRae & Hetherington, 1993; Sloman & Rumelhart, 1992).”
Self-refreshing memory in artificial neural networks: learning temporal sequences without catastrophic forgetting (2004, background): “disastrous for practical applications, which is why numerous authors over the last decade and a half have developed ways to overcome this problem (Hetherington and Seidenberg 1989, McCloskey and Cohen 1989, Kortge 1990, Ratcliff 1990, Lewandowsky 1991, 1994, Murre 1992, French 1992, 1994, 1997, Mc”
Catastrophic Forgetting and the Pseudorehearsal Solution in Hopfield-type Networks (1998, methodology): “This problem has been identified in many forms in memory literature – it is commonly called the “catastrophic forgetting”, “catastrophic interference”, or “serial learning” problem (Hetherington and Seidenberg, 1989; McCloskey and Cohen, 1989; Lewandowsky, 1991; French, 1992; McRae and Hetherington,”
Pseudo-recurrent Connectionist Networks: An Approach to the 'Sensitivity-Stability' Dilemma (1997, methodology): “In previous work on catastrophic forgetting (French, 1992, 1994; Murre, 1992; etc.) these are exactly the type of representations that has been suggested were needed to reduce representational overlap, thereby reducing the interference that led to severe forgetting.”
Sequential learning in neural networks: A review and a discussion of pseudorehearsal based methods (2004, unknown): “Robins and Frean [56] discussed the relationship between pseudorehearsal and other local learning methods such as RBF networks, activation sharpening [17] and kNN. Pseudorehearsal was shown to be signiﬁcantly more effective than the other local methods tested on a serial learning task.”
AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence (2019, background): “An example type of problem that could in theory be solved with meta-learning and neuromodulation is catastrophic forgetting, a phenomenon wherein neural networks are unable to continuously learn because they learn each newly encountered task by overwriting their knowledge of how to solve all previou”
The Division of Labor Between the Neocortex and Hippocampus (2010, background): “A number of researchers showed that this interference can be reduced by introducing various factors that result in less pattern overlap (e.g., Kortge, 1993; French, 1992; Sloman & Rumelhart, 1992; McRae & Hetherington, 1993).”
Continual Learning in Recurrent Neural Networks with Hypernetworks (2020, unknown): “Lastly, some regularization approaches rely on the use of non-overlapping and orthogonal representations to overcome catastrophic forgetting [25, 26, 27].”
Neural networks with a self-refreshing memory: Knowledge transfer in sequential learning tasks without catastrophic forgetting (2000, background): “…Taylor & Francis Ltd http://www.tandf.co.uk/journals 1989, McCloskey and Cohen 1989, Kortge 1990, Ratcliff 1990, Lewandowsky 1991, 1994, Murre 1992, French 1992, 1994, 1997, McRae and Hetherington 1993, Lewandowsky and Li 1995, McClelland et al. 1995, Sharkey and Sharkey 1995, Robins 1995,…”