Overcoming Catastrophic Interference using Conceptor-Aided Backpropagation

By Xu He, Herbert Jaeger. 2018

128 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Meta & Continual Learning

Citations in the corpus

Continual Lifelong Learning with Neural Networks: A Review (2018, positive): “ence, the two approaches have shown similar results on the permuted MNIST benchmark (LeCun et al., 1998; see Fig. 3.a), whereas experiments on more complex and larger datasets have not been reported. He and Jaeger (2018) introduced conceptor-aided backpropagation (CAB), which uses conceptors to shie”
Synaptic weight decay with selective consolidation enables fast learning without catastrophic forgetting (2019, background): “A yet more global criterion for learning rate adaptation has been suggested that prevents catastrophic interference using conceptors that seek to orthogonalize the pattern representation in diﬀerent contexts (He and Jaeger, 2018).”
Continual Training of Language Models for Few-Shot Learning (2022, methodology): “(2) Replay methods retain (Rebufﬁ et al., 2017; Lopez-Paz and Ranzato, 2017; Wang et al., 2020; Guo et al., 2022) or generate some data of old tasks (Shin et al., 2017; He and Jaeger, 2018) and use them in learning a new task.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “In image classification, He and Jaeger [19] replaced the typical backpropagation with conceptor-aided backpropagation.”
Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks (2021, methodology): “The methods in (Shin et al., 2017; Kamra et al., 2017; Rostami et al., 2019; He and Jaeger, 2018) build data generators for previous tasks so that in learning the new task, they can use some generated data for previous tasks to help avoid forgetting.”
Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning (2021, methodology): “This is done by either incorporating additional penalty terms into the loss function [33, 1, 62, 48] or modifying the gradient of parameters during optimization [36, 8, 22].”
Online Continual Learning in Image Classification: An Empirical Survey (2021, methodology): “This is done by either incorporating additional penalty terms into the loss function [35, 36, 37, 38] or modifying the gradient of parameters during optimization [13, 31, 39].”
Linear Mode Connectivity in Multitask and Continual Learning (2020, background): “In this sense it connects to topics such as MetaLearning (Beaulieu et al., 2020; Jerfel et al., 2019; He & Jaeger, 2018; Riemer et al., 2018), Few-Shot Learning (Wen et al., 2018; Gidaris & Komodakis, 2018), Multi-task and Transfer Learning (He & Jaeger, 2018; Jerfel et al., 2019).”
Understanding the Role of Training Regimes in Continual Learning (2020, methodology): “With a frequentist view, some other regularization based methods have utilized gradient information to protect previous knowledge [14, 24, 79].”
Orthogonal Gradient Descent for Continual Learning (2019, methodology): “He and Jaeger (2018) proposed a variant of the back-propagation algorithm named conceptor-aided backprop that shields gradients against degradation of previously learned tasks.”
Continual Unsupervised Representation Learning (2019, methodology): “In a similar vein, other methods encourage new tasks to utilise previously unused parameters, either by ﬁnding “free” linear parameter subspaces (He & Jaeger, 2018); learning an attention mask over parameters (Serra et al., 2018); or using an agent to ﬁnd new activation paths through a network…”
Continual Learning by Asymmetric Loss Approximation With Single-Side Overestimation (2019, background): “An interesting work among the architectural approaches, instead of adding new network components, employs a network compression technique to identify unused or rarely used parts in the target network and allows the free space to store the information of new tasks [25, 12].”
Continual learning with hypernetworks (2019, methodology): “Despite relying on regularization, our approach is a conceptual departure from previous algorithms based on regularization in weight (e.g., Kirkpatrick et al., 2017; Zenke et al., 2017) or activation space (e.g., He & Jaeger, 2018).”
Continual Learning for Sentence Representations Using Conceptors (2019, background): “Conceptors have their origin in reservoir computing (Jaeger, 2014) and recently have been used to perform continual learning in deep neural networks (He and Jaeger, 2018).”