SPeCiaL: Self-Supervised Pretraining for Continual Learning

By Lucas Caccia, Joelle Pineau. 2021

20 citations recorded by the source corpus.

This paper presents SPeCiaL: a method for unsupervised pretraining of representations tailored for continual learning. Our approach devises a meta-learning objective that differentiates through a sequential learning process. Specifically, we train a linear model over the representations to match different augmented views of the same image together, each view presented sequentially. The linear model is then evaluated on both its ability to classify images it just saw, and also on images from previous iterations. This gives rise to representations that favor quick knowledge retention with minimal forgetting. We evaluate SPeCiaL in the Continual Few-Shot Learning setting, and show that it can match or outperform other supervised pretraining approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Continual Barlow Twins: Continual Self-Supervised Learning for Remote Sensing Semantic Segmentation (2022, background): “We note that in CV, the combination of these two technologies is starting to be explored, and some works have already started to demonstrate how SSL methods are well prepared, after small modifications, to learn incrementally [7], [10].”
Self-Supervised Models are Continual Learners (2021, background): “Recently, [5,23] explored self-supervised pretraining for supervised continual learning with online and few-shot tasks, and [10] presented a supervised contrastive CL approach.”
Continual Learning via Local Module Composition (2021, background): “Continual-meta learning is often confused with its counterpart meta-continual learning [40, 11, 93], in which algorithm are learning to continually learn.”
Continual Learning on a Diet: Learning from Sparsely Labeled Streams Under Constrained Computation (2024, background): “Caccia & Pineau (2022) showed that self-supervised loss outperformed supervised loss in the meta-learning stage for continual learning.”
Continual Learning with Foundation Models: An Empirical Study of Latent Replay (2022, methodology): “…on a large external dataset ( e.g. ImageNet (Deng et al., 2009)), except speciﬁc applications ( e.g. segmentation (Cermelli et al., 2020), meta CL (Caccia and Pineau, 2021; Caccia et al., 2020; Javed and White, 2019)), is rarely used in CL. Hayes et al. (2018); Lesort et al. (2021b); Chrysakis…”