Avoiding catastrophic forgetting by coupling two reverberating neural networks

By L’oubli catastrophique it, deuX rkeaux, neuronaux verb-ants. 2004

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Toward Understanding Catastrophic Forgetting in Continual Learning (2019, background): “However, continual learning by deep models has proven to be very challenging due to catastrophic forgetting , a long known problem of training deep neural networks [4, 5, 14, 15, 28, 31, 34].”
Continual Learning with Deep Generative Replay (2017, background): “When the tasks are as elementary as coupling two binary patterns, simply feeding random noises and corresponding responses suffices [2].”
iCaRL: Incremental Classifier and Representation Learning (2016, background): “…standard classification loss, which encourages improvements of the feature representation that allow classifying the newly observed classes well, it also contains the distillation loss, which ensures that the discriminative information learned previously is not lost during the new learning step.”
FearNet: Brain-Inspired Model for Incremental Learning (2017, methodology): “GeppNet is the most recent DNN to be based on this theory, but it was also independently explored in the 1990s in French (1997) and Ans & Rousset (1997).”
The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects (2013, background): “Inorder to prevent catastrophic forgetting, various researchers have suggested using a dual-memory system which, fundamentally, simulates the presence of a short-term and a long-term memory (Robins, 1995; Ans and Rousset, 1997; French, 1997; Mermillod et al., 2003).”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, positive): “While there have been several models inspired by CLS theory (Gepperth & Karaoguz, 2016; French, 1997; Ans & Rousset, 1997; Kemker & Kanan, 2018; Robins, 1995), many existing replay approaches have only focused on modeling medial prefrontal cortex directly and do not have a fast learning network.”
Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent (2020, methodology): “Several heuristics were developed in order to characterise it (Ans & Rousset 1997, Ans & Rousset 2000, Goodfellow et al. 2014, French 1999, McCloskey & Cohen 1989, Robins 1995, Nguyen et al. 2019).”
Learning to Learn without Forgetting By Maximizing Transfer and Minimizing Interference (2018, background): “A second strategy known as dual network models (McClelland et al., 1995; Ans & Rousset, 1997) is based on the neurobiological finding that both hippocampal and cortical circuits contributed differentially to memory.”
Continual learning: A comparative study on how to defy forgetting in classification tasks (2019, background): “Early research works [22], [23], [24] developed several strategies to mitigate the forgetting under the condition of not storing the training data, mostly at a small scale of just a few examples and considering shallow networks.”
Selfless Sequential Learning (2018, background): “Since the early development of neural networks, researchers have faced the catastrophic forgetting problem and developed several strategies to mitigate the forgetting under the condition of not storing the previous tasks training data [20, 4, 34].”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “Two early examples include pseudo recurrent networks (French, 1997) and coupling two reverberating neural networks (Ans and Rousset, 1997).”
Self-refreshing memory in artificial neural networks: learning temporal sequences without catastrophic forgetting (2004, methodology): “Each input–output pair of vectors produced in this way will constitute a pseu-dopattern that will be a reﬂection of the function previously learned by the network.”
Exemplar-Supported Generative Reproduction for Class Incremental Learning (2018, background): “Practical approaches include dual-memory models [2, 12], evolving networks [18, 31], weight consolidation [15], and other variants of neural networks [4, 23].”
A biologically inspired dual-network memory model for reduction of catastrophic forgetting (2014, methodology): “To evaluate a network0s ability to correctly reproduce the appropriate outputs for a given set of inputs, a performance measure, called goodness [3,4,16], was adopted.”
Mitigation of catastrophic forgetting in recurrent neural networks using a Fixed Expansion Layer (2013, methodology): “Dual network methods use two neural networks in order to separate the task of learning from the task of long-term information storage (e.g. [12], [1], [7]).”
Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgetting (2018, methodology): “These methods are further extended by separating the learning agent into two parts, an idea which stems from early research into CF [14], [15], [16] and is still being utilised in recent algorithms such as Deep Generative Dual Memory Network [17] for incremental image classification and others (e.”
Neuron Clustering for Mitigating Catastrophic Forgetting in Supervised and Reinforcement Learning (2015, background): “In the past, substantial research was aimed at addressing catastrophic forgetting [10] [11] [12] [13] [14].”
Sequential learning in neural networks: A review and a discussion of pseudorehearsal based methods (2004, unknown): “…number of data sets, including the Iris data used in [52] and summarised above;autoassociative and heteroassociative randomly constructed data sets [1,51]; classiﬁcation tasks using the Mushroom and Congressional Voting Records data sets (see [44]) and a Cat and Dog picture dataset [19,20];…”
The contents of long-term memory and the emergence of knowledge (2009, background): “It is also central to connectionist models (e.g., Ans & Rousset, 1997, 2000; McClelland & Rumelhart, 1986; for a review, see, for example, Hintzman, 1990), in which the traces are by definition distributed across all the connections in the network.”