Continual semi-supervised learning through contrastive interpolation consistency

By Matteo Boschini, Pietro Buzzega, Lorenzo Bonicelli, Angelo Porrello, Simone Calderara. 2021

51 citations recorded by the source corpus.

Continual Learning (CL) investigates how to train Deep Networks on a stream of tasks without incurring forgetting. CL settings proposed in literature assume that every incoming example is paired with ground-truth annotations. However, this clashes with many real-world applications: gathering labeled data, which is in itself tedious and expensive, becomes infeasible when data flow as a stream. This work explores Continual Semi-Supervised Learning (CSSL): here, only a small fraction of labeled input examples are shown to the learner. We assess how current CL methods (e.g.: EWC, LwF, iCaRL, ER, GDumb, DER) perform in this novel and challenging scenario, where overfitting entangles forgetting. Subsequently, we design a novel CSSL method that exploits metric learning and consistency regularization to leverage unlabeled examples while learning. We show that our proposal exhibits higher resilience to diminishing supervision and, even more surprisingly, relying only on 25% supervision suffices to outperform SOTA methods trained under full supervision.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Subsequent work includes training an adversarial autoencoder to reconstruct images [265], imposing predictive consistency among augmented and interpolated data [266], and leveraging the nearest-neighbor classiﬁer to distill class-instance relationships [267].”
A soft nearest-neighbor framework for continual semi-supervised learning (2022, neutral): “, the use of external datasets, the accessibility of labeled/unlabeled data during continual learning stages, leaving only a few [8, 53] to be comparable in”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, methodology): “To assess our proposal, we perform a suite of experiments through the Mammoth framework [12, 17, 9, 6, 27, 13, 14, 50], an open-source codebase introduced in [15] for testing CL algorithms.”
Transfer without Forgetting (2022, background): “On a final note, recent works [7,61] exploit the memory buffer to address semi-supervised settings where examples can be either labeled or not.”
CaSpeR: Latent Spectral Regularization for Continual Learning (2023, background): “…classes seen so far, the limited memory buffer size produces various overfitting issues that constitute the focus of many recent works (e.g., abruptly divergent gradients for new classes [11, 9], dete-riorating decision surface [6], accumulation of predictive bias for current classes [48, 1]).”
Continual Learning on a Diet: Learning from Sparsely Labeled Streams Under Constrained Computation (2024, methodology): “Gomez-Villa et al. (2022); Pham et al. (2021); Boschini et al. (2022) used unlabeled data for distillation or regularization loss to migrate forgetting.”