Self-Supervised Training Enhances Online Continual Learning

By Jhair Gallardo, Tyler L. Hayes, Christopher Kanan. 2021

74 citations recorded by the source corpus.

In continual learning, a system must incrementally learn from a non-stationary data stream without catastrophic forgetting. Recently, multiple methods have been devised for incrementally learning classes on large-scale image classification tasks, such as ImageNet. State-of-the-art continual learning methods use an initial supervised pre-training phase, in which the first 10% - 50% of the classes in a dataset are used to learn representations in an offline manner before continual learning of new classes begins. We hypothesize that self-supervised pre-training could yield features that generalize better than supervised learning, especially when the number of samples used for pre-training is small. We test this hypothesis using the self-supervised MoCo-V2, Barlow Twins, and SwAV algorithms. On ImageNet, we find that these methods outperform supervised pre-training considerably for online continual learning, and the gains are larger when fewer samples are available. Our findings are consistent across three online continual learning algorithms. Our best system achieves a 14.95% relative increase in top-1 accuracy on class incremental ImageNet over the prior state of the art for online continual learning.

Subfields: Education, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Education

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “In addition to an earlier work on obtaining sparse representations through meta-training [185], recent works have attempted to incorporate the advantages of selfsupervised learning (SSL) [125], [281], [335] and large-scale pre-training [295], [380], [456] to improve the representations in initializa”
Self-Supervised Models are Continual Learners (2021, background): “Recently, [5,23] explored self-supervised pretraining for supervised continual learning with online and few-shot tasks, and [10] presented a supervised contrastive CL approach.”
Co2L: Contrastive Continual Learning (2021, background): “Concurrent works [43, 15, 30] exploit self-supervised learning objectives to learn more generalizable representations than ones trained with supervised learning objectives.”
Continual Learning with Foundation Models: An Empirical Study of Latent Replay (2022, background): “Finally, echoing Gallardo et al. (2021), Hu et al. (2022) showed that SSL was also important for external pre-training.”
Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2 (2022, methodology): “, 2019a;b)) and self-supervised learning (Gallardo et al., 2021).”
Task Agnostic Representation Consolidation: a Self-supervised based Continual Learning Approach (2022, methodology): “Gallardo et al. (2021) empirically showed that self-supervised pre-training yields representations that generalize better across tasks than supervised pre-training in CL. Owing to additional computational effort, some of the approaches, e.g. (Zhang et al., 2020; Caccia et al., 2021; Mazumder et…”
Online Continual Learning for Embedded Devices (2022, methodology): “For replay, we follow Gallardo et al. (2021) We include the final top-1 accuracies achieved by each online continual learning method with each CNN backbone on the Places-365 and Places-LT datasets in Table 4 and Table 5, respectively.”