Self-Supervised Models are Continual Learners

By Enrico Fini, Victor Costa, Xavier Alameda-Pineda, Elisa Ricci, Alahari Karteek, Julien Mairal. 2021

236 citations recorded by the source corpus.

Self-supervised models have been shown to produce comparable or better visual representations than their su-pervised counterparts when trained offline on unlabeled data at scale. However, their efficacy is catastrophically reduced in a Continual Learning (CL) scenario where data is presented to the model sequentially. In this paper, we show that self-supervised loss functions can be seamlessly converted into distillation mechanisms for CL by adding a predictor network that maps the current state of the repre-sentations to their past state. This enables us to devise a framework for Continual self-supervised visual representation Learning that (i) significantly improves the quality of the learned representations, (ii) is compatible with several state-of-the-art self-supervised objectives, and (iii) needs little to no hyperparameter tuning. We demonstrate the ef-fectiveness of our approach empirically by training six pop-ular self-supervised models in various CL settings. Code: github.com/DonkeyShot21/cassle.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “CaSSLe [195] converts the self-supervised loss to a distillation strategy by mapping the current state of a representation to its previous state.”
SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model (2023, background): “On the other hand, since the large amount of pre-training data are typically unlabeled and may also arrive incrementally, it seems more reasonable to use self-supervised pre-training than supervised pre-training [5, 40], also regarding that (upstream) continual learning in a self-supervised manner i”
Towards Label-Efficient Incremental Learning: A Survey (2023, methodology): “Finally, self-supervised learners omit the need for label-supervision by designing pretext tasks from the incremental learning tasks [Fini et al., 2022].”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, neutral): “To assess our proposal, we perform a suite of experiments through the Mammoth framework [12, 17, 9, 6, 27, 13, 14, 50], an open-source codebase introduced in [15] for testing CL algorithms.”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “For CORe50 and DomainNet, we follow [66] and [18] to present the average forward classiﬁcation accuracy.”
Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replay (2022, background): “Besides, inspired by the remarkable progress in self-supervised visual representation learning [1,2,6,7,19], [12] propose to enable continual learning without labeled data.”
Continual Barlow Twins: Continual Self-Supervised Learning for Remote Sensing Semantic Segmentation (2022, background): “We note that in CV, the combination of these two technologies is starting to be explored, and some works have already started to demonstrate how SSL methods are well prepared, after small modifications, to learn incrementally [7], [10].”
Continual Pre-Training Mitigates Forgetting in Language and Vision (2022, methodology): “Unlike recent studies (Fini et al., 2022; Hu et al., 2021; Jang et al., 2021; Jin et al., 2022; Ke et al., 2023-02-01), we do not employ any custom CL strategy neither during pre-training nor during fine-tuning.”
Continual Learning, Fast and Slow (2021, methodology): “Concurrent with our work, several studies have explored incorporating SSL into continual learning and achieved promising results [54], [55], [56].”
Consistency is the key to further mitigating catastrophic forgetting in continual learning (2022, background): “Enforcing consistency in predictions separated through time helps in preserving rich information about the previous tasks better thereby helping to further mitigate the catastrophic forgetting in CL.”
Continual Learning on a Diet: Learning from Sparsely Labeled Streams Under Constrained Computation (2024, methodology): “Then, we report the accuracy of the union test sets in the last time step T as A ( T ) = a T following Chaudhry et al. (2019a); Fini et al. (2022); Wang et al. (2022).”
Scaling the Number of Tasks in Continual Learning (2022, background): “In the literature, other works used indirect methods such as linear probing to witness that forgetting in deep neural networks is maybe not as catastrophic as expected Fini et al. [2022], Davari et al. [2022].”
Is Multi-Task Learning an Upper Bound for Continual Learning? (2022, methodology): “To avoid catastrophic forgetting, we use the CaSSLe framework [19], which is designed for continual SSL.”
Online Prototype Learning for Online Continual Learning (2023, methodology): “PCL is an unsupervised contrastive learning method while OPE explicitly leverages class labels to compute online prototypes.”
Continual Self-Supervised Learning: Towards Universal Multi-Modal Medical Data Representation Learning (2023, methodology): “These CL techniques can be summarized in five main categories, including regularization-based approach [25], rehearsal-based approach [5, 35], optimization-based approach [28], representation-based approach [14], and architecture-based approach [32].”