Learning a Unified Classifier Incrementally via Rebalancing
By Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, Dahua Lin. 2019
1,402 citations recorded by the source corpus.
Conventionally, deep neural networks are trained offline, relying on a large dataset prepared in advance. This paradigm is often challenged in real-world applications, e.g. online services that involve continuous streams of incoming data. Recently, incremental learning receives increasing attention, and is considered as a promising solution to the practical challenges mentioned above. However, it has been observed that incremental learning is subject to a fundamental difficulty -- catastrophic forgetting, namely adapting a model to new data often results in severe performance degradation on previous tasks or classes. Our study reveals that the imbalance between previous and new data is a crucial cause to this problem. In this work, we develop a new framework for incrementally learning a unified classifier, e.g. a classifier that treats both old and new classes uniformly. Specifically, we incorporate three components, cosine normalization, less-forget constraint, and inter-class separation, to mitigate the adverse effects of the imbalance. Experiments show that the proposed method can effectively rebalance the training process, thus obtaining superior performance compared to the existing methods. On CIFAR-100 and ImageNet, our method can reduce the classification errors by more than 6% and 13% respectively, under the incremental setting of 10 phases.
Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Radiology, Nuclear Medicine and Imaging
- De-chuan Zhan
- Han-Jia Ye
- Da-Wei Zhou
- Bernt Schiele
- Christopher Kanan
- Tyler L. Hayes
- Joost van de Weijer
- Matthieu Cord
In Meta & Continual Learning
- Han-Jia Ye
- De-chuan Zhan
- Da-Wei Zhou
- Joost van de Weijer
- Liyuan Wang
- Jun Zhu
- Bing Liu
- Xingxing Zhang
Citations in the corpus
- A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “stability, emergent strategies include renormalization of the old and new task solutions [166], [263], [384], [442], balanced exploitation of the old and new training samples [10], [167], [174], [452], [469], [513], reserving space for subsequent tasks [9], [199], [518], etc.”
- S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning (2023, methodology): “UCIR [16] proposed cosine-based classifiers and used feature-space distillation and inter-class separation margin loss to mitigate catastrophic forgetting.”
- Learnability and Algorithm for Continual Learning (2023, methodology): “…and jointly train the model using the current data and the saved buffer data (Rusu et al., 2016; LopezPaz & Ranzato, 2017; Chaudhry et al., 2019a; Hou et al., 2019; Wu et al., 2019; Rolnick et al., 2019; Buzzega et al., 2020; Rajasegaran et al., 2020a; Liu et al., 2021; Cha et al., 2021; Yan et…”
- Task-Aware Information Routing from Common Representation Space in Lifelong Learning (2023, background): “Task-recency bias: CL models trained in an incremental learning scenario tend to be biased towards the most recent tasks, termed task-recency bias (Hou et al., 2019).”
- RMM: Reinforced Memory Management for Class-Incremental Learning (2023, methodology): “We follow the benchmark protocol used in [13, 18, 30, 31].”
- CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, background): “Rehearsal with stored data [3–5, 7, 9, 10, 18, 19, 24, 29, 42, 51–53, 64, 68] or samples from a generative model [27, 28,48,56,60] is highly effective when storing training data or training/saving a generative model is possible.”
- Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, methodology): “Comparison Methods To demonstrate the efficacy of our method, we adopt ER [9], UCIR [21], GeoDL [41] as comparison methods.”
- On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, neutral): “However, recent works put an increasing focus on the harder Class-Incremental setting (CIL) [4, 32, 15, 81], which entails the production of a unified prediction encompassing all seen classes.”
- Continual Learning by Modeling Intra-Class Variation (2022, methodology): “The last type of method, activation-based regularization (Li & Hoiem, 2017; Rebuffi et al., 2017; Hou et al., 2019; Wu et al., 2019; Yu et al., 2020), leverages the information from the activations obtained from past tasks using strategies similar to knowledge distillation (Hinton et al., 2015).”
- Margin-Based Few-Shot Class-Incremental Learning with Class-Level Overfitting Mitigation (2022, neutral): “Few-shot class-incremental learning (FSCIL) can be roughly divided into finetune-based methods [16, 26, 4, 31] and metric-based methods [40, 44].”
- A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learning (2022, methodology): “LUCIR [9] proposed to use a balanced fine-tuning to train the classifier after freezing feature representation.”
- Few-Shot Continual Active Learning by a Robot (2022, background): “Most continual machine learning models [7, 8, 9, 10], however, are developed for constrained task-based continual learning setups, where a CL model continually learns a sequence of tasks, one at a time, with all the data of the current task labeled and available in an increment.”
- Long-Tailed Class Incremental Learning (2022, methodology): “In this section we analyze the performance of four popular CIL methods: LwF [20] with exempalrs, EEIL [7], LUCIR [13], and PODNET [10]).”
- Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “Specifically, on CIFAR100, adding CSCCT to iCARL [42], LUCIR [19] and PODNet [13] provides strong relative improvement of 2.76%, 2.28% and 1.99% respectively averaged across all settings, while on the much more highdimensional ImageNet-Subset, adding our method to the respective baselines provides c”
- Few-Shot Class-Incremental Learning from an Open-Set Perspective (2022, neutral): “We compare our method with the SOTA methods [17,1,11,19,4,5,23] on three datasets.”