SupportNet: solving catastrophic forgetting in class incremental learning with support data
By Yu Li, Zhongxiao Li, Lizhong Ding, Peng Yang, Yuhui Hu, Xin Gao. 2018
31 citations recorded by the source corpus.
A plain well-trained deep learning model often does not have the ability to learn new knowledge without forgetting the previously learned knowledge, which is known as catastrophic forgetting. Here we propose a novel method, SupportNet, to efficiently and effectively solve the catastrophic forgetting problem in the class incremental learning scenario. SupportNet combines the strength of deep learning and support vector machine (SVM), where SVM is used to identify the support data from the old data, which are fed to the deep learning model together with the new data for further training so that the model can review the essential information of the old data when learning the new information. Two powerful consolidation regularizers are applied to stabilize the learned representation and ensure the robustness of the learned model. We validate our method with comprehensive experiments on various tasks, which show that SupportNet drastically outperforms the state-of-the-art incremental learning methods and even reaches similar performance as the deep learning model trained from scratch on both old and new data. Our program is accessible at: this https URL
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
Citations in the corpus
- Do not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting (2021, methodology): “Other data-based incremental methods are proposed in [24].”
- Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting (2019, methodology): “Early attempts to alleviate catastrophic forgetting often consists of memory system that store previous data and replay the sampled old examples with the new data (Robins, 1995), and similar approaches are still used today (Rebuffi et al., 2017b; Li et al., 2018; Lopez-Paz et al., 2017).”
- Overcoming Catastrophic Forgetting With Unlabeled Data in the Wild (2019, background): “model can only see the given labeled training dataset during training [3, 12, 23, 24, 33].”
- Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data (2018, methodology): “To determine the most important samples for the previous classes, the approach in [24] trains a support vector machine (SVM) from the previous training data.”
- M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning (2019, methodology): “Our approach also works for exemplar based incremental learning which use distillation sequentially on the output of networks [25, 2, 16].”
- Deep learning in bioinformatics: introduction, application, and perspective in big data era (2019, methodology): “If we only fine-tune the model with the new data, the fine-tuned model’s performance on the older classes will be unacceptable [89].”
- Machine learning for discovering missing or wrong protein function annotations (2019, methodology): “However, the previous version of these datasets should not be disregarded, since having two versions can be of interest to perform an evaluation similar to ours on new HMC methods, or to other fields inmachine learning such as weakly supervised classification, noise detection and incremental learnin”
- Overcoming Catastrophic Forgetting by Bayesian Generative Regularization (2019, methodology): “Previous works also try to alleviate catastrophic forgetting by introducing memory systems which store previous data and replay the stored old examples with the new data (Farquhar & Gal, 2019; Li et al., 2018; Lopez-Paz & Ranzato, 2017; Rebuffi et al., 2017; Robins, 1995).”
- Overcoming Catastrophic Forgetting by Generative Regularization (2019, methodology): “Generative Models Previous works also try to alleviate catastrophic forgetting by introducing memory systems which store previous data and replay the stored old examples with the new data (Robins, 1995; Rebuffi et al., 2017; Li et al., 2018; Lopez-Paz & Ranzato, 2017).”