Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning

By Zheda Mai, Ruiwen Li, Hyunwoo J. Kim, Scott Sanner. 2021

237 citations recorded by the source corpus.

Online class-incremental continual learning (CL) studies the problem of learning new classes continually from an online non-stationary data stream, intending to adapt to new data while mitigating catastrophic forgetting. While memory replay has shown promising results, the recency bias in online learning caused by the commonly used Softmax classifier remains an unsolved challenge. Although the Nearest-Class-Mean (NCM) classifier is significantly undervalued in the CL community, we demonstrate that it is a simple yet effective substitute for the Softmax classifier. It addresses the recency bias and avoids structural changes in the fully-connected layer for new classes. Moreover, we observe considerable and consistent performance gains when replacing the Softmax classifier with the NCM classifier for several state-of-the-art replay methods.To leverage the NCM classifier more effectively, data embeddings belonging to the same class should be clustered and well-separated from those with a different class label. To this end, we contribute Supervised Contrastive Replay (SCR), which explicitly encourages samples from the same class to cluster tightly in embedding space while pushing those of different classes further apart during replay-based training. Overall, we observe that our proposed SCR substantially reduces catastrophic forgetting and outperforms state-of-the-art CL methods by a significant margin on a variety of datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “OCL InstAParam [68], CBRS [75], ASER [382], InfoRS [398], DVC [143], CTN [336], NCCL [475], ILOS [158], SCR [284], CVT [441], OCM [145], ER-ACE [47], RAR [498], PoLRS [50], AOP [144], CV [159]”
Repeated Augmented Rehearsal: A Simple but Strong Baseline for Online Continual Learning (2022, methodology): “Each mini batch during training consists of 10 new and 10 memory samples, except for the SCR method, which employs 100 memory samples and 10 new incoming samples Mai et al. [2021].”
Few-Shot Class-Incremental Learning from an Open-Set Perspective (2022, neutral): “As real-world classification problems typically exhibit class imbalance or long-tailed data distribution, some methods have explored deep metric learning for incremental and long-tail tasks [22,15,12].”
Online Continual Learning with Contrastive Vision Transformer (2022, methodology): “SCR [52] uses supervised contrastive loss for representation learning and employs the nearest class mean to classify.”
Co2L: Contrastive Continual Learning (2021, background): “Concurrent works [43, 15, 30] exploit self-supervised learning objectives to learn more generalizable representations than ones trained with supervised learning objectives.”
New Insights on Reducing Abrupt Representation Change in Online Continual Learning (2021, background): “D. Finally, concurrent work (Mai et al., 2021) also use a contrastive loss for online continual learning, but not in an asymmetric fashion.”
Improving Plasticity in Online Continual Learning via Collaborative Learning (2023, methodology): “Following [31], we remove the final FC classifier and use Nearest-Class-Mean (NCM) [31] classifier with intermediate representations.”
CaSpeR: Latent Spectral Regularization for Continual Learning (2023, methodology): “SCR [30] enforces consistency between two views of the input batch by leveraging the Supervised Contrastive loss [24]; differently, CSCCT [4] pairs an explicit latent-space clustering objec-tive that prevents the dispersion of previously-learned class boundaries with a controlled transfer objective…”
Class-Incremental Learning: A Survey (2023, methodology): “Online CIL : Although data comes with stream format, the model can conduct multi-epoch training with each task, i.e. , offline training within each task.”
Online Continual Learning through Mutual Information Maximization (2022, methodology): “SCR (Mai et al., 2021) uses supervised contrastive loss for representation learning and the nearest class mean for classification.”
Retrospective Adversarial Replay for Continual Learning (2022, background): “This category also includes memory-based approaches for constrained parameter updates on new samples to minimize interference with previous tasks [9, 37] and for replaying them along with the new task’s data [2, 3, 5, 6, 10, 13, 38, 41, 43, 44, 47, 48, 59, 55, 65].”
Consistent Representation Learning for Continual Relation Extraction (2022, methodology): “In the testing phase, the nearest class mean (NCM) classiﬁer is used to classify the test sample, which will not be affected by the deviation of the classiﬁer.”
Online Prototype Learning for Online Continual Learning (2023, methodology): “In addition, for DER++[4], SCR [44], and DVC [25], we follow their default settings and use their own extra data augmentations.”
Deep Class-Incremental Learning: A Survey (2023, background): “There are some works addressing fully online (one-pass) class-incremental learning, where each batch can be processed once and then dropped [37], [38].”
Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation (2024, methodology): “SCR [40] takes supervised contrastive loss and nearest-class-mean classifier.”
Dealing with Cross-Task Class Discrimination in Online Continual Learning (2023, unknown): “GSA 2 is compared with 16 recent baselines: 9 online CL baselines, AGEM , ER , MIR , GSS , ASER , RainBow-rt , ER-AML , GDumb , and SCR and 7 offline CL baselines, DER++ , IL2A , Co 2 L , LUCIR , CCIL , BiC and SSIL , as they deal with data/class imbalance and can be run in the online CL mode, i.e.,”
Computationally Budgeted Continual Learning: What Does Matter? (2023, unknown): “SupCon SCR [32] Reservoir Random SupCon NCM”