Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer

By James Smith, Jonathan C. Balloch, Yen-Chang Hsu, Zsolt Kira. 2021

49 citations recorded by the source corpus.

Rehearsal is a critical component for class-incremental continual learning, yet it requires a substantial memory budget. Our work investigates whether we can significantly reduce this memory budget by leveraging unlabeled data from an agent's environment in a realistic and challenging continual learning paradigm. Specifically, we explore and formalize a novel semi-supervised continual learning (SSCL) setting, where labeled data is scarce yet non-i.i.d. unlabeled data from the agent's environment is plentiful. Importantly, data distributions in the SSCL setting are realistic and therefore reflect object class correlations between, and among, the labeled and unlabeled data distributions. We show that a strategy built on pseudo-labeling, consistency regularization, Out-of- Distribution (OoD) detection, and knowledge distillation reduces forgetting in this setting. Our approach, DistillMatch, increases performance over the state-of-the-art by no less than 8.7% average task accuracy and up to 54.5% average task accuracy in SSCL CIFAR-100 experiments. Moreover, we demonstrate that DistillMatch can save up to 0.23 stored images per processed unlabeled image compared to the next best method which only saves 0.08. Our results suggest that focusing on realistic correlated distributions is a significantly new perspective, which accentuates the importance of leveraging the world's structure as a continual learning strategy. Our code is available at https://github.com/GT-RIPL/DistillMatch-SSCL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Meta & Continual Learning

Citations in the corpus

Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, neutral): “SETS is completely different from CL and SSCL in several aspects: (i) our SETS focuses on a large-scale task sequence (more than 72K), while CL and SSCL are not scalable to this large-scale setting; (ii) our goal is to enable the model to generalize to the unseen tasks from all the previous task dis”
Transfer without Forgetting (2022, background): “On a final note, recent works [7,61] exploit the memory buffer to address semi-supervised settings where examples can be either labeled or not.”
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, methodology): “Critically, these methods cannot be used in applications with privacy concerns [53] or when memory budget is highly constrained [54].”
Continual Sequence Generation with Adaptive Compositional Modules (2022, background): “…follows a line of work that doesn’t store real examples for experience replay, such as generating examples by GAN (Atkinson et al., 2018), synthesizing examples (Xu et al., 2022) by model-inversion (Smith et al., 2021b), and using unlabeled data in the learning environment (Smith et al., 2021a).”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, positive): “Smith et al. [215] combined regularization and knowledge distillation based methods.”
Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning (2021, neutral): “, penalize changes to model predictions) [7, 23, 33, 36, 52].”
Hypernetworks for Continual Semi-Supervised Learning (2021, unknown): “However, the unlabeled data in continual tasks signiﬁcantly differ from our set-up, as [Smith et al. , 2021] uses the super-class structure of the CIFAR dataset.”