Dynamically constraining connectionist networks to produce distributed, orthogonal representations to reduce catastrophic interference

By R. M. French. 2019

66 citations recorded by the source corpus.

It is well known that when a connectionist network is trained on one set of patterns and then attempts to add new patterns to its repertoire, catastrophic interference may result. The use of sparse, orthogonal hidden-layer representations has been shown to reduce catastrophic interference. The author demonstrates that the use of sparse representations may, in certain cases, actually result in worse performance on catastrophic interference. This paper argues for the necessity of maintaining hidden-layer representations that are both as highly distributed and as highly orthogonal as possible. The author presents a learning algorithm, called context-biasing, that dynamically solves the problem of constraining hiddenlayer representations to simultaneously produce good orthogonality and distributedness. On the data tested for this study, context-biasing is shown to reduce catastrophic interference by more than 50% compared to standard backpropagation. In particular, this technique succeeds in reducing catastrophic interference on data where sparse, orthogonal distributions failed to produce any improvement.

Subfields: Artificial Intelligence, Meta & Continual Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Meta & Continual Learning

In Signal Processing

Citations in the corpus

Deep Generative Dual Memory Network for Continual Learning (2017, methodology): “Following this, subsequent works attempted to remedy the issue by reducing representational overlap between input representations via activation sharpening algorithms (Kortge, 1990), orthogonal recoding of inputs (Lewandowsky, 1991) or orthogonal activations at all hidden layers (McRae & Hetheringto”
Few-Shot Continual Active Learning by a Robot (2022, background): “In recent years, several works in CL have focused on mitigating the catastrophic forgetting problem [13, 3, 6].”
End-to-End Incremental Learning (2018, neutral): “Initial attempts to overcome this issue were aimed at connectionist networks [2, 8, 18], and are thus inapplicable in the context of today’s deep architectures for computer vision problems.”
Incremental Learning of Object Detectors without Catastrophic Forgetting (2017, background): “This phenomenon of forgetting is believed to be caused by two factors [11, 22].”
Continual learning: A comparative study on how to defy forgetting in classification tasks (2019, background): “Several directions have been explored, such as reducing representation overlap [22], [34], [35], [36], [37], replaying samples or virtual samples from the past [10], [24] or introducing dual architectures [38], [39], [40].”
Selfless Sequential Learning (2018, methodology): “This has led to several methods with the goal of orthogonalizing the activations (French, 1992; 1994; Kruschke, 1992; 1993; Sloman & Rumelhart, 1992).”
Storing Encoded Episodes as Concepts for Continual Learning (2020, background): “) One of the main problems faced by class-incremental learning approaches is catastrophic forgetting (French, 2019; Kirkpatrick et al., 2017), in which the model forgets the previously learned classes when learning new classes and the overall classification accuracy decreases.”
Brain-Inspired Model for Incremental Learning Using a Few Examples (2020, background): “Essentially, classification accuracy rapidly decreases when learning new classes [2, 13, 15, 21, 24, 29].”
Tell me what this is: Few-Shot Incremental Object Learning by a Robot (2020, background): “Attempts to create deep models that learn incrementally have been hindered by a problem known as catastrophic forgetting [2].”
Learning Novel Objects Continually Through Curiosity (2021, background): “Yet, a deep neural network requires a large corpus of high-quality labeled data in a single batch [6],”
Online Learning of Objects through Curiosity-Driven Active Learning (2021, background): “Yet, a deep neural network requires a large corpus of high-quality labeled data in a single batch [6], which makes it impractical for real-world robotics applications.”
Cognitively-Inspired Model for Incremental Learning Using a Few Examples (2020, background): “Essentially, clas-siﬁcation accuracy rapidly decreases when learning new classes [2, 13, 16, 22, 25, 30].”
Online Task-free Continual Learning with Dynamic Sparse Distributed Memory (2022, methodology): “This was done in several ways, either by gradually nudging the network’s internal representations into largely non-overlapping clusters [49] or by orthogonalizing the input representations [15, 33, 35].”
Incremental Detection of Remote Sensing Objects With Feature Pyramid and Knowledge Distillation (2020, background): “This phenomenon of forgetting is caused by two main factors [28], [29].”
Incremental learning with self-organizing maps (2017, background): “As the problem of catastrophic forgetting was first remarked for multilayer perceptron (MLP) models [4], [5], it is hardly surprising that there is significant work on the subject of how catastrophic forgetting could be avoided[16], [17], [18], [19], [20], [21], [22], [23] although none of these pro”