ScaIL: Classifier Weights Scaling for Class Incremental Learning

By Eden Belouadah, Adrian Daniel Popescu. 2020

90 citations recorded by the source corpus.

Incremental learning is useful if an AI agent needs to integrate data from a stream. The problem is non trivial if the agent runs on a limited computational budget and has a bounded memory of past data. In a deep learning approach, the constant computational budget requires the use of a fixed architecture for all incremental states. The bounded memory generates imbalance in favor of new classes and a prediction bias toward them appears. This bias is commonly countered by introducing a data balancing step in addition to the basic network training. We depart from this approach and propose simple but efficient scaling of past classifiers’ weights to make them more comparable to those of new classes. Scaling exploits incremental state statistics and is applied to the classifiers learned in the initial state of classes to profit from all their available data. We also question the utility of the widely used distillation loss component of incremental learning algorithms by comparing it to vanilla fine tuning in presence of a bounded memory. Evaluation is done against competitive baselines using four public datasets. Results show that the classifier weights scaling and the removal of the distillation are both beneficial.

Subfields: Artificial Intelligence, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, neutral): “Label Space LwF [252], iCaRL [356], GEM [277], A-GEM [65], EEIL [53], BiC [457], WA [500], DER++ [46], ScaIL [33], S&B [218], SS-IL [11], Coil [506]”
S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning (2023, methodology): “Several state-of-art-works [38,45,2,12,3] proposed different techniques to address the class imbalance problem in CIL like rescaling scores or balanced finetuning of classifiers, etc.”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, methodology): “In computer vision, several researchers used the term incremental learning to mean Class-IL as they learn like any other Class-IL method with multiple classes per task (Wu et al. 2019; Castro et al. 2018; Liu et al. 2020; Lee et al. 2019; Belouadah and Popescu 2019, 2020).”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, positive): “They used selective memory retrieval, which is shown to be more effective compared to random retrieval in continual analogical reasoning.”
Generalized and Incremental Few-Shot Learning by Explicit Learning and Calibration without Forgetting (2021, background): “On the other hand, some work addresses it with bias removal techniques [49, 3, 4] by training batch of additional hyperparameters [49], using dual memory [3] or with post-processing [4].”
Continual Learning by Using Information of Each Class Holistically (2021, background): “One typical approach is to memorize some exemplars of old tasks to perform knowledge distillation (Wu et al. 2019; Castro et al. 2018; Liu et al. 2020; Lee et al. 2019; Belouadah and Popescu 2019, 2020) to deal with CF.”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, positive): “For example, several of the most successful models for incremental learning store a subset of previously learned raw inputs in a replay buffer (Gepperth and Karaoguz, 2016; Rebuffi et al., 2017; Lopez-Paz and Ranzato, 2017; Castro et al., 2018; Chaudhry et al., 2018; Nguyen et al., 2018; Hou et al.,”
Online Continual Learning in Image Classification: An Empirical Survey (2021, neutral): “This also confirms the results of three recent studies, where [110] shows the shortcomings of regularization-based approaches in the class incremental setting, [111] theoretically explains why regularization-based methods underperform memory-based methods and [112] empirically demonstrates that KD i”
Do not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting (2021, background): “ScaIL [2] discusses an efficient scaling of past classifiers’ weights to make them more comparable to those of new classes in incremental learning scenarios.”
PLOP: Learning without Forgetting for Continual Semantic Segmentation (2020, background): “Other works focus on adaptive architectures that can extend themselves to integrate new classes [67, 40] or dynamically re-arrange coexisting sub-networks [20] each specialized in one specific task [19, 23, 32], or to explicitly correct the classifier drift [66, 74, 3, 4] that happens with continual”
SS-IL: Separated Softmax for Incremental Learning (2020, methodology): “As mentioned in the Introduction, several previous works [8, 23, 16, 31, 4, 35, 5] identified that the major challenge of the exemplar-memory based CIL is to resolve the classification score bias caused by the data imbalance.”
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020, methodology): “The dominant approach is based on model updating via fine-tuning to integrate new knowledge [5, 6, 10, 19, 58].”
Dataset Condensation with Differentiable Siamese Augmentation (2021, background): “The coreset selection competitors are random, herding (Chen et al., 2010; Rebuffi et al., 2017; Castro et al., 2018; Belouadah & Popescu, 2020) and forgetting (Toneva et al., 2019).”
Memory-Efficient Class-Incremental Learning for Image Classification (2020, unknown): “BiC [51] adds a linear model after the last fully connected layer to correct bias toward new classes and ScaIL [57] scales past classifiers’ weights to make them more comparable to those of new classes.”
Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning (2021, background): “Besides, several approaches are learning with memorized data (Castro et al., 2018; Javed & Shafait, 2018; Rebuffi et al., 2017; Belouadah & Popescu, 2019; 2020; Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018).”
Queried Unlabeled Data Improves and Robustifies Class-Incremental Learning (2022, unknown): “On the one hand, existing methods that store and replay past training data (Castro et al., 2018; Javed & Shafait, 2018; Rebuﬃ et al., 2017; Belouadah & Popescu, 2019; 2020) are still the most eﬀective in overcoming catastrophic forgetting, despite its storage headache and sometimes the data…”
Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation (2021, background): “It can be also possible to fix the knowledge drift in the classifier by recalibration [47], [48], [49], [50].”
Initial Classifier Weights Replay for Memoryless Class Incremental Learning (2020, methodology): “Note that features from the current IL state were used successfully with initial classifiers in [5] if a bounded memory of the past was allowed.”
Dataset Condensation with Distribution Matching (2021, methodology): “training set condensation to the random selection that is used in [35], herding [12, 36, 10, 4] and DSA [51].”