Attention-Based Structural-Plasticity

By Soheil Kolouri, Nicholas A. Ketz, Xinyun Zou, J. Krichmar, Praveen K. Pilly. 2019

12 citations recorded by the source corpus.

Catastrophic forgetting/interference is a critical problem for lifelong learning machines, which impedes the agents from maintaining their previously learned knowledge while learning new tasks. Neural networks, in particular, suffer plenty from the catastrophic forgetting phenomenon. Recently there has been several efforts towards overcoming catastrophic forgetting in neural networks. Here, we propose a biologically inspired method toward overcoming catastrophic forgetting. Specifically, we define an attention-based selective plasticity of synapses based on the cholinergic neuromodulatory system in the brain. We define synaptic importance parameters in addition to synaptic weights and then use Hebbian learning in parallel with backpropagation algorithm to learn synaptic importances in an online and seamless manner. We test our proposed method on benchmark tasks including the Permuted MNIST and the Split MNIST problems and show competitive performance compared to the state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mehrdad Farajtabar
2. Seyed Iman Mirzadeh
3. Razvan Pascanu
4. J. Urgen Schmidhuber
5. Sjoerd van Steenkiste
6. Ang Li
7. Dilan Gorur
8. Hassan Ghasemzadeh

In Computer Vision and Pattern Recognition

1. Ling Shao
2. Mehrdad Farajtabar
3. Luc Van Gool
4. Dacheng Tao
5. Ming-Hsuan Yang
6. Xiaogang Wang
7. Trevor Darrell
8. Seyed Iman Mirzadeh

In Electrical and Electronic Engineering

1. Ling Shao
2. Hassan Ghasemzadeh
3. Seyed Iman Mirzadeh
4. J. Urgen Schmidhuber
5. Cees G. M. Snoek
6. Ang Li
7. Mehrdad Farajtabar
8. Razvan Pascanu

Citations in the corpus

• Kernel Continual Learning (2021, methodology): “Regularization methods (Kirkpatrick et al., 2017; Aljundi et al., 2018; Lee et al., 2017; Zenke et al., 2017; Kolouri et al., 2019) determine the importance of each model’s parameter for each task, which prevents the parameters from being updated for new tasks.”
• Linear Mode Connectivity in Multitask and Continual Learning (2020, methodology): “Regularization: These methods explicitly apply regularization techniques to ensure parameters do not change too much (Kirkpatrick et al., 2017; Zenke et al., 2017; Lee et al., 2017; Aljundi et al., 2018; Kolouri et al., 2019).”
• Orthogonal Gradient Descent for Continual Learning (2019, methodology): “Other works along this line are (Aljundi et al., 2018; Kolouri et al., 2019) which penalize the weights based on a Hebbian like update rule.”
• Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks (2020, methodology): “Here, we will measure the amount of transfer in a continual learning setup using the popular permuted MNIST benchmark (Kirkpatrick et al., 2017; Golkar et al., 2019; Kolouri et al., 2019).”
• Continual Learning with Filter Atom Swapping (2022, background): “(Kirkpatrick et al., 2017b; Aljundi et al., 2018a; Lee et al., 2017; Zenke et al., 2017b; Kolouri et al., 2019) determine the importance of each model’s parameter per task, which prevents the important parameters from being updated for new tasks.”