Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control

By Inyoung Paik, Sangjun Oh, Taeyeong Kwak, Injung Kim. 2019

65 citations recorded by the source corpus.

To address the issue of catastrophic forgetting in neural networks, we propose a novel, simple, and effective solution called neuron-level plasticity control (NPC). While learning a new task, the proposed method preserves the existing knowledge from the previous tasks by controlling the plasticity of the network at the neuron level. NPC estimates the importance value of each neuron and consolidates important neurons by applying lower learning rates, rather than restricting individual connection weights to stay close to the values optimized for the previous tasks. The experimental results on the several datasets show that neuron-level consolidation is substantially more effective compared to connection-level consolidation approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hang Su
2. Kyle Richardson
3. Jun Zhu
4. Liyuan Wang
5. Xingxing Zhang
6. Ashish Sabharwal
7. Xiaorong Li
8. Jian Sun

In Computer Vision and Pattern Recognition

1. Hang Su
2. Jun Zhu
3. Kyle Richardson
4. Liyuan Wang
5. Xingxing Zhang
6. Zongben Xu
7. Ashish Sabharwal
8. Xiaorong Li

In Meta & Continual Learning

1. Xingxing Zhang
2. Liyuan Wang
3. Hang Su
4. Jun Zhu
5. Zongben Xu
6. Simone Scardapane
7. Valerio Marsocci
8. Fei Ye

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Instead of consolidating parameters, NPC [114] estimates the importance of each neuron and selectively reduces its learning rate.”
• Training Networks in Null Space of Feature Covariance for Continual Learning (2021, methodology): “The typical methods include EWC [22], SI [55], MAS [3], RWalk [6] and NPC [39].”
• What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge (2019, background): “…knowledge (Peters et al., 2019); we see many areas where our inoculation strategy could be improved for such purposes, including having more complex loss functions that manage old and new information, as well as using techniques that take into account network plasticity (Paik et al., 2019).”