Large Scale Incremental Learning
By Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, Yun Raymond Fu. 2019
1,535 citations recorded by the source corpus.
Modern machine learning suffers from \textit{catastrophic forgetting} when learning new classes incrementally. The performance dramatically degrades due to the missing data of old classes. Incremental learning methods have been proposed to retain the knowledge acquired from the old classes, by using knowledge distilling and keeping a few exemplars from the old classes. However, these methods struggle to \textbf{scale up to a large number of classes}. We believe this is because of the combination of two factors: (a) the data imbalance between the old and new classes, and (b) the increasing number of visually similar classes. Distinguishing between an increasing number of visually similar classes is particularly challenging, when the training data is unbalanced. We propose a simple and effective method to address this data imbalance issue. We found that the last fully connected layer has a strong bias towards the new classes, and this bias can be corrected by a linear model. With two bias parameters, our method performs remarkably well on two large datasets: ImageNet (1000 classes) and MS-Celeb-1M (10000 classes), outperforming the state-of-the-art algorithms by 11.1\% and 13.2\% respectively.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Han-Jia Ye
- De-chuan Zhan
- Bernt Schiele
- Jun Zhu
- Dacheng Tao
- Joost van de Weijer
- Mohit Bansal
- Yoshua Bengio
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Bernt Schiele
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Yi Yang
- Andrea Vedaldi
In Meta & Continual Learning
- Han-Jia Ye
- De-chuan Zhan
- Da-Wei Zhou
- Joost van de Weijer
- Liyuan Wang
- Jun Zhu
- Xingxing Zhang
- Bing Liu
Citations in the corpus
- A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “BiC [147] and WA [148] attribute this issue to the bias of the last fully-connectedlayer,andresolveitbyeitherlearningabiascorrection layerwithabalancedvalidationset[147],ornormalizingthe outputweights[148].”
- S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning (2023, methodology): “Several state-of-art-works [38,45,2,12,3] proposed different techniques to address the class imbalance problem in CIL like rescaling scores or balanced finetuning of classifiers, etc.”
- Learnability and Algorithm for Continual Learning (2023, methodology): “…the model using the current data and the saved buffer data (Rusu et al., 2016; LopezPaz & Ranzato, 2017; Chaudhry et al., 2019a; Hou et al., 2019; Wu et al., 2019; Rolnick et al., 2019; Buzzega et al., 2020; Rajasegaran et al., 2020a; Liu et al., 2021; Cha et al., 2021; Yan et al., 2021; Wang et…”
- First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning (2023, background): “While softmax cross entropy loss is considered a standard in classification models, it is shown to be a source of class interference in continual learning [38, 4].”
- SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model (2023, methodology): “As for the former, a memory buffer of 1000 images is maintained, and we evaluate three representative replay-based approaches such as BiC [49], GDumb [30] and DER++ [3].”
- RMM: Reinforced Memory Management for Class-Incremental Learning (2023, methodology): “…task-incremental learning—each phase corresponds to a task (dataset) that contains new data of all seen classes [7, 10, 19, 29, 43, 47, 57], or class-incremental learning (CIL)—each phase contains data of a new set of classes, i.e., classes are unseen [4, 6, 18, 25, 31, 37, 40, 41, 49, 52, 55–57].”
- A soft nearest-neighbor framework for continual semi-supervised learning (2022, neutral): “The first category comprises regularization-based methods [13,14,20,22,24,25,30,45,54], which address the problem of catastrophic forgetting by introducing appropriate regularization terms in the objective function or identifying a set of parameters that are most relevant for certain tasks.”
- FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning (2022, methodology): “Many recent solutions [4, 13, 33, 44, 46], based on deep nets, use replay from a bounded memory of the past to reduce forgetting.”
- CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, methodology): “We also compare to three rehearsalbased methods to provide additional context to our results: Experience Replay [11], Bias Correction (BiC) [67], and Dark-Experience Replay++ (DERP++) [7].”
- A Theoretical Study on Solving Continual Learning (2022, methodology): “The inconsistency of LwF is due to its severe classification bias towards later tasks as discussed in BiC [32].”
- Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, methodology): “In particular, those methods [52,47] aiming to alleviate class imbalance are not applicable to the continual learning of VLP models as there are no classlevel supervision for VLP.”
- ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation (2022, methodology): “On the contrary, class-incremental methods [4, 8, 33, 36] attempt to recognize all categories observed so far at once without the task identifier, being more practical.”
- On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, neutral): “However, recent works put an increasing focus on the harder Class-Incremental setting (CIL) [4, 32, 15, 81], which entails the production of a unified prediction encompassing all seen classes.”
- Continual Learning by Modeling Intra-Class Variation (2022, methodology): “The last type of method, activation-based regularization (Li & Hoiem, 2017; Rebuffi et al., 2017; Hou et al., 2019; Wu et al., 2019; Yu et al., 2020), leverages the information from the activations obtained from past tasks using strategies similar to knowledge distillation (Hinton et al., 2015).”
- A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learning (2022, methodology): “We also consider popular continual learning methods including regularization-based methods (LwF[13] , EWC [10] ) and rehearsal-based methods (ER[4], GDumb[19], BiC [29], DER++ [1] and Co2L [3]).”
- Few-Shot Continual Active Learning by a Robot (2022, background): “Most continual machine learning models [7, 8, 9, 10], however, are developed for constrained task-based continual learning setups, where a CL model continually learns a sequence of tasks, one at a time, with all the data of the current task labeled and available in an increment.”
- Long-Tailed Class Incremental Learning (2022, methodology): “Replay-based methods are very effective and recall knowledge from previous tasks by maintaining a small memory of samples [2,4,7,13,32,41], representations [12], or synthetic data [35].”
- Repeated Augmented Rehearsal: A Simple but Strong Baseline for Online Continual Learning (2022, background): “Therefore, it allows training on each single task with multiple epochs (e.g., 70-200 epochs) [Rebuffi et al., 2017, Wu et al., 2019, Cha et al., 2021].”
- Anti-Retroactive Interference for Lifelong Learning (2022, methodology): “5, We compare different methods on CUB200 with 6 incremental tasks where BiC [30] and CoIL [37] are memory-based methods.”
- Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “It is not possible to use the drifting feature space of FTt to represent Cnew; even representing Cnew once it has been learned by FTt would heavily bias the representations towards Cnew due to the well-known recency bias [51, 2].”