SS-IL: Separated Softmax for Incremental Learning

By Hongjoon Ahn, Jihwan Kwak, Ser-Nam Lim, Hyeonsu Bang, Taesup Moon. 2020

219 citations recorded by the source corpus.

We consider class incremental learning (CIL) problem, in which a learning agent continuously learns new classes from incrementally arriving training data batches and aims to predict well on all the classes learned so far. The main challenge of the problem is the catastrophic forgetting, and for the exemplar-memory based CIL methods, it is generally known that the forgetting is commonly caused by the classification score bias that is injected due to the data imbalance between the new classes and the old classes (in the exemplar-memory). While several methods have been proposed to correct such score bias by some additional post-processing, e.g., score re-scaling or balanced fine-tuning, no systematic analysis on the root cause of such bias has been done. To that end, we analyze that computing the softmax probabilities by combining the output scores for all old and new classes could be the main cause of the bias. Then, we propose a new method, dubbed as Separated Softmax for Incremental Learning (SS-IL), that consists of separated softmax (SS) output layer combined with task-wise knowledge distillation (TKD) to resolve such bias. Throughout our extensive experimental results on several large-scale CIL benchmark datasets, we show our SS-IL achieves strong state-of-the-art accuracy through attaining much more balanced prediction scores across old and new classes, without any additional post-processing.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, neutral): “…tasksolutions[52],[104],[113],[154],[196],balancedexploita-tionofoldandnewtrainingsamples[46],[129],[147],[148], [149],[152],spacereservationforsubsequenttasks[115],[116], [256],etc. Ontheotherhand,solutiongeneralizabilitycould…”
First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning (2023, background): “Thus, recent works either focus on fixing the softmax classifier [4, 1, 44] or deploy nearest class mean classifiers (NCM) as an alternative [25, 5, 11].”
CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, background): “We note that this bias is a well-known issue [1, 11].”
Continual Learning by Modeling Intra-Class Variation (2022, methodology): “For online continual learning, the compared approaches include: A-GEM (Chaudhry et al., 2018), an online version of GEM, MIR (Aljundi et al., 2019a), a buffer selection method considering the sample influence to performance and SS-IL (Ahn et al., 2021), a method separating the old-class and new-clas”
Long-Tailed Class Incremental Learning (2022, methodology): “Replay-based methods are very effective and recall knowledge from previous tasks by maintaining a small memory of samples [2,4,7,13,32,41], representations [12], or synthetic data [35].”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “SS-IL [2] show that general KD preserves the recency bias in the distillation, and propose to use task-wise KD.”
Class-Incremental Learning with Strong Pre-trained Models (2022, background): “For the first group, the general approach for incremental training is to impose constraints on parameter changes [3, 13, 32], gradients [17], features [8,11,25], or activations [2,6,10,15,23] when finetuning the network with novel samples.”
Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation (2022, methodology): “For comparisons, we report average incremental accuracy [37]—the average of the accuracies over seen classes across all incremental stages, where the results of other methods except GeoDL [44], DDE [17], and SSIL [2] are retrieved from [12] if not specified otherwise.”
Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “Our proposed representation compensation module can be easily integrated with many existing continual learning methods [2, 29, 89].”
Class-Incremental Continual Learning Into the eXtended DER-Verse (2022, neutral): “As also observed in other recent works [28], [29], [42], this issue can be mitigated by revising the way the cross-entropy loss is applied during training.”
Generalized and Incremental Few-Shot Learning by Explicit Learning and Calibration without Forgetting (2021, background): “…when we apply smaller (30) number of epochs during the second phase and compare it to a longer second phase (150 epochs). shorten the second Due to the negligible differences we use the initially chosen value that equals 150 epochs throughout the main paper. phase 0.2 points higher accuracy”
SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning (2021, methodology): “…the seminal work, [27], most of the recent neural network-based CIL research has focused on the classiﬁcation setting, and various approaches have been proposed to address the main challenge of the problem, the so-called plasticity-stability dilemma, e.g. , [20, 30, 12, 1, 2, 8], to name a few.”
New Insights on Reducing Abrupt Representation Change in Online Continual Learning (2021, methodology): “This property emerges from only learning the incoming data in isolation ; as we will see, also isolating the rehearsal step (as in Ahn et al. (2020)) leads to poor knowledge acquisition on the current task.”
Essentials for Class Incremental Learning (2021, methodology): “only using intra-task learning module as proposed in [1]-v1.”
Deep Class-Incremental Learning: A Survey (2023, methodology): “There are other works [75], [77], [82], [92], [104], [116], [139] utilizing MNIST [191], Core50 [192], Google Landmarkv2 [193], CUB200-2011 [194], TinyImageNet [195], and miniImageNet [190] for model evaluation, while the aforementioned datasets are the most widely adopted in the current CIL communi”