Overcoming Catastrophic Interference by Conceptors

By Xu He, Herbert Jaeger. 2017

14 citations recorded by the source corpus.

Catastrophic interference has been a major roadblock in the research of continual learning. Here we propose a variant of the back-propagation algorithm, "conceptor-aided back-prop" (CAB), in which gradients are shielded by conceptors against degradation of previously learned tasks. Conceptors have their origin in reservoir computing, where they have been previously shown to overcome catastrophic forgetting. CAB extends these results to deep feedforward networks. On the disjoint MNIST task CAB outperforms two other methods for coping with catastrophic interference that have recently been proposed in the deep learning field.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Razvan Pascanu
2. Mehrdad Farajtabar
3. Bing Liu
4. Seyed Iman Mirzadeh
5. Dacheng Tao
6. Tinne Tuytelaars
7. Yoshua Bengio
8. Sergey Levine

In Statistical and Nonlinear Physics

1. Mehrdad Farajtabar
2. Razvan Pascanu
3. Seyed Iman Mirzadeh
4. Hassan Ghasemzadeh
5. Yee Whye Teh
6. Wojciech M. Czarnecki
7. Joan Serrà
8. Raia Hadsell

In Representation & Self-Supervised Learning

1. Razvan Pascanu
2. Bing Liu
3. Tinne Tuytelaars
4. Mehrdad Farajtabar
5. Joost van de Weijer
6. Seyed Iman Mirzadeh
7. Hassan Ghasemzadeh
8. Yee Whye Teh

In Meta & Continual Learning

1. Mehrdad Farajtabar
2. Seyed Iman Mirzadeh
3. Razvan Pascanu
4. Hassan Ghasemzadeh
5. Dilan Gorur
6. Yee Whye Teh
7. Raia Hadsell
8. Agnieszka Grabska-Barwinska

Citations in the corpus

• Overcoming Multi-Model Forgetting (2019, positive): “He & Jaeger (2017) project the gradient so that directions relevant to the previous task are unaffected.”
• Overcoming catastrophic forgetting with hard attention to the task (2018, background): “In this setting, the best result from the literature corresponds to the conceptor-aided backpropagation approach (He & Jaeger, 2017), with A ≤ 2 = 94 .”
• Progress & Compress: A scalable framework for continual learning (2018, background): “Recently He & Jaeger (2018) proposed a different mechanism, which employs a projection of the gradients such that no direction relevant to the previous task is affected.”
• Linear Mode Connectivity in Multitask and Continual Learning (2020, background): “The orthogonality has also been tried as a hard constraint in a number of continual learning papers explicitly (Farajtabar et al., 2019; He & Jaeger, 2017).”
• Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization (2018, background): “To our knowledge, these numbers of tasks are an order of magnitude greater compared to previously published work (Zenke et al., 2017; Kirkpatrick et al., 2017; He and Jaeger, 2017; Nguyen et al., 2017; Serrà et al., 2018).”
• Physical reservoir computing—an introductory perspective (2020, unknown): “[38–40]).”