Rectification-based Knowledge Retention for Continual Learning

By Pravendra Singh, Pratik Mazumder, Piyush Rai, Vinay P. Namboodiri. 2021

51 citations recorded by the source corpus.

Deep learning models suffer from catastrophic forgetting when trained in an incremental learning setting. In this work, we propose a novel approach to address the task incremental learning problem, which involves training a model on new tasks that arrive in an incremental manner. The task incremental learning problem becomes even more challenging when the test set contains classes that are not part of the train set, i.e., a task incremental generalized zero-shot learning problem. Our approach can be used in both the zero-shot and non zero-shot task incremental learning settings. Our proposed method uses weight rectifications and affine transformations in order to adapt the model to different tasks that arrive sequentially. Specifically, we adapt the network weights to work for new tasks by "rectifying" the weights learned from the previous task. We learn these weight rectifications using very few parameters. We additionally learn affine transformations on the outputs generated by the network in order to better adapt them for the new task. We perform experiments on several datasets in both zero-shot and non zero-shot task incremental learning settings and empirically show that our approach achieves state-of-the-art results. Specifically, our approach outperforms the state-of-the-art non zero-shot task incremental learning method by over 5% on the CIFAR-100 dataset. Our approach also significantly outperforms the state-of-the-art task incremental generalized zero-shot learning method by absolute margins of 6.91% and 6.33% for the AWA1 and CUB datasets, respectively. We validate our approach using various ablation studies.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Hang Su
5. Jun Zhu
6. Liyuan Wang
7. Xingxing Zhang
8. Ziwei Liu

In Computer Vision and Pattern Recognition

1. Han-Jia Ye
2. De-chuan Zhan
3. Jun Zhu
4. Da-Wei Zhou
5. Hang Su
6. Ziwei Liu
7. Dacheng Tao
8. Liyuan Wang

In Radiology, Nuclear Medicine and Imaging

1. Da-Wei Zhou
2. De-chuan Zhan
3. Han-Jia Ye
4. Yajing Kong
5. Liyuan Wang
6. Dacheng Tao
7. Jun Zhu
8. Mingg-Ming Cheng

In Meta & Continual Learning

1. Xingxing Zhang
2. Liyuan Wang
3. Han-Jia Ye
4. Hang Su
5. De-chuan Zhan
6. Jun Zhu
7. Da-Wei Zhou
8. Qiwen Wang

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “According to the provided Xt and Yt in each Dt, continual learning can be further categorized into zero-shot [132], [388], few-shot [404], semi-supervised [428], open-world (i.”
• Online Continual Learning with Contrastive Vision Transformer (2022, background): “As new tasks arrive, expansion-based methods dynamically expand networks and keep sub-networks related to previous tasks fixed [69,89,1,59,86,45,82,53].”
• Representation Compensation Networks for Continual Semantic Segmentation (2022, methodology): “Some other methods [45,57,75,76,82,91] increase the capacity of the model to have a better tradeoff of stability and plasticity, but with the cost of growing memory of the network.”
• Deep Class-Incremental Learning: A Survey (2023, unknown): “2 Model Rectify Feature Rectify [106], [107], [108], [109], [110], [111]”
• Continual Learning with Lifelong Vision Transformer (2022, background): “Most of the early works in continual learning focus on the task-incremental learning (task-IL), where oracle knowledge of the task identity is available at inference time for selecting the corresponding classifier [2, 17, 44, 65, 68].”
• Class-Incremental Learning: A Survey (2023, background): “RKR [109] proposes to rectify the convolutional weights of old models when learning new tasks.”