Residual Continual Learning

By Janghyeon Lee, Donggyu Joo, Hyeong Gwon Hong, Junmo Kim. 2020

25 citations recorded by the source corpus.

We propose a novel continual learning method called Residual Continual Learning (ResCL). Our method can prevent the catastrophic forgetting phenomenon in sequential learning of multiple tasks, without any source task information except the original network. ResCL reparameterizes network parameters by linearly combining each layer of the original network and a fine-tuned network; therefore, the size of the network does not increase at all. To apply the proposed method to general convolutional neural networks, the effects of batch normalization layers are also considered. By utilizing residual-learning-like reparameterization and a special weight decay loss, the trade-off between source and target performance is effectively controlled. The proposed method exhibits state-of-the-art performance in various continual learning scenarios.

Subfields: Artificial Intelligence, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Zhu
2. Hang Su
3. Bing Liu
4. Xingxing Zhang
5. Jitendra Malik
6. Liyuan Wang
7. Leonidas J. Guibas
8. Amir Zamir

In Meta & Continual Learning

1. Bing Liu
2. Xingxing Zhang
3. Liyuan Wang
4. Qi Qin
5. Hang Su
6. Jun Zhu
7. Wenpeng Hu
8. Mengyu Wang

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “ResCL [112] extends this idea with a learnable combination coefficient.”
• Continual Learning by Using Information of Each Class Holistically (2021, methodology): “Inspired by (Lee et al. 2017, 2020), we propose a post-processing step to eliminate the Shared Knowledge to reduce their negative effects on classification in testing (no change to training).”
• Knowledge Capture and Replay for Continual Learning (2020, neutral): “, 2019], [Lee et al., 2020] [Ebrahimi et al.”
• Residual Tuning: Toward Novel Category Discovery Without Labels (2022, unknown): “However, previous works like RCL [61] and Side-tuning [22], increase the number of network parameters by a large margin, due to creating a whole network without sharing the layers.”
• Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks (2019, methodology): “Some network adaptation methods insert new residual-modeling parameters directly into the base architecture [14,24].”