Co2L: Contrastive Continual Learning

By Hyuntak Cha, Jaeho Lee, Jinwoo Shin. 2021

408 citations recorded by the source corpus.

Recent breakthroughs in self-supervised learning show that such algorithms learn visual representations that can be transferred better to unseen tasks than cross-entropy based methods which rely on task-specific supervision. In this paper, we found that the similar holds in the continual learning context: contrastively learned representations are more robust against the catastrophic forgetting than ones trained with the cross-entropy objective. Based on this novel observation, we propose a rehearsal-based continual learning algorithm that focuses on continually learning and maintaining transferable representations. More specifically, the proposed scheme (1) learns representations using the contrastive learning objective, and (2) preserves learned representations using a self-supervised distillation step. We conduct extensive experimental validations under popular benchmark image classification datasets, where our method sets the new state-of-the-art performance. Source code is available at https://github.com/chaht01/Co2L.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Co2L [92] adopts a supervised contrastive loss to learn each task and a self-supervised loss to distill knowledge between the old and new models.”
Task-Aware Information Routing from Common Representation Space in Lifelong Learning (2023, methodology): “Following Arani et al. (2022); Buzzega et al. (2020); Cha et al. (2021); Caccia et al. (2021a), we employ ResNet-18 (He et al., 2016) as the backbone to learn a common representation space for all our experiments.”
A soft nearest-neighbor framework for continual semi-supervised learning (2022, neutral): “The first category comprises regularization-based methods [13,14,20,22,24,25,30,45,54], which address the problem of catastrophic forgetting by introducing appropriate regularization terms in the objective function or identifying a set of parameters that are most relevant for certain tasks.”
A Theoretical Study on Solving Continual Learning (2022, methodology): “[37] Hyuntak Cha, Jaeho Lee, and Jinwoo Shin.”
Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, methodology): “Co(2)L[8] introduces a self-supervised knowledge distillation method for self-supervised continual learning.”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, positive): “While rehearsal strategies are by far the most frequently adopted thanks to their effectiveness and flexibility [26, 4], it is not uncommon to adopt solutions combining multiple approaches [32, 1, 15, 60, 18].”
Class Gradient Projection For Continual Learning (2022, methodology): “Another method [6] combines the contrastive learning with continual learning.”
A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learning (2022, methodology): “We also consider popular continual learning methods including regularization-based methods (LwF[13] , EWC [10] ) and rehearsal-based methods (ER[4], GDumb[19], BiC [29], DER++ [1] and Co2L [3]).”
Repeated Augmented Rehearsal: A Simple but Strong Baseline for Online Continual Learning (2022, background): “Therefore, it allows training on each single task with multiple epochs (e.g., 70-200 epochs) [Rebufﬁ et al., 2017, Wu et al., 2019, Cha et al., 2021].”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “With respect to clustering, an important point is that our loss is cross-space in the sense, it does not encourage clustering of features of a class using features from the same model [8], that would neither exploits prior knowledge about the inter-class distances, nor imposes any constraints on the”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “They include non-prompting methods EWC [32], LwF [40] ER [8], GDumb [53], BiC [67], DER++ [4] and Co2L [6], as well as the prompting methods DyTox [14] and L2P [66] and self-supervised learning method CaSSLe [18].”
Transfer without Forgetting (2022, background): “– Contrastive Continual Learning (CO(2)L) [11] proposes to facilitate knowledge transfer from samples stored in the buffer by optimizing a contrastive learning objective, avoiding any potential bias introduced by a cross-entropy objective.”
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, methodology): “ER [7, 13], GDumb [45], BiC [62], DER++ [3] and Co(2)L [4].”
Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “The knowledge can be stored by multiple types, like examples [4, 6, 9, 11, 73, 83], prototypes [35, 106, 107], generative networks [60], etc.”
Learning to Prompt for Continual Learning (2021, methodology): “However, the performance of rehearsal-based methods generally deteriorates with smaller buffer size [4], and rehearsal-based methods are eventually not applicable to scenarios where data privacy should be taken into account [54].”
Self-Supervised Models are Continual Learners (2021, background): “The former [34] extends [10] to the unsupervised setting, but is specifically designed for contrastive SSL, such as [13, 27], and lacks generalizability to other popular SSL paradigms.”
A Contrastive Continual Learning for the Classification of Remote Sensing Imagery (2022, methodology): “Taking inspiration from the work of Hyuntak Cha et al. in [13] where they argue that learning good features can help combating catastrophic forgetting in a continual learning scenario.”
Online Continual Learning through Mutual Information Maximization (2022, background): “3 are recent replay-based batch CL baselines that use the knowledge distillation loss by running them in one epoch: DER++ (Buzzega et al., 2020), IL2A (Zhu et al., 2021), Co 2 L (Cha et al., 2021).”