Representational Continuity for Unsupervised Continual Learning

By Divyam Madaan, Jaehong Yoon, Yuanchun Li, Yunxin Liu, Sung Ju Hwang. 2021

152 citations recorded by the source corpus.

Continual learning (CL) aims to learn a sequence of tasks without forgetting the previously acquired knowledge. However, recent CL advances are restricted to supervised continual learning (SCL) scenarios. Consequently, they are not scalable to real-world applications where the data distribution is often biased and unannotated. In this work, we focus on unsupervised continual learning (UCL), where we learn the feature representations on an unlabelled sequence of tasks and show that reliance on annotated data is not necessary for continual learning. We conduct a systematic study analyzing the learned feature representations and show that unsupervised visual representations are surprisingly more robust to catastrophic forgetting, consistently achieve better performance, and generalize better to out-of-distribution tasks than SCL. Furthermore, we find that UCL achieves a smoother loss landscape through qualitative analysis of the learned representations and learns meaningful feature representations. Additionally, we propose Lifelong Unsupervised Mixup (LUMP), a simple yet effective technique that interpolates between the current task and previous tasks' instances to alleviate catastrophic forgetting for unsupervised representations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Observing that self-supervised representations are more robust to catastrophic forgetting, LUMP [93] acquires further improvements by interpolating between instances of the old and new tasks.”
Towards Label-Efficient Incremental Learning: A Survey (2023, methodology): “Formally, following the notation in [Madaan et al. , 2022], the learner receives a sequence of learning tasks T 1: t = ( T 1 , T 2 , ..., T t ) , with a corresponding dataset D T = { ( x i,t , y i,t ) n t i =1 } with n t instances.”
Task-Free Continual Learning via Online Discrepancy Distance Learning (2022, neutral): “Another kind of approaches for continual learning focuses on the memory system, which usually employs a small memory buffer [3, 5, 6, 28, 36, 44, 59] for storing some past data or trains a generator to provide the replay samples when learning new tasks [38, 43, 46, 47, 58, 57, 52].”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, neutral): “To assess our proposal, we perform a suite of experiments through the Mammoth framework [12, 17, 9, 6, 27, 13, 14, 50], an open-source codebase introduced in [15] for testing CL algorithms.”
Consistency is the key to further mitigating catastrophic forgetting in continual learning (2022, background): “Intersection of SSL and CL: There have been several works (Madaan et al., 2021; Fini et al., 2021; Davari et al., 2022) that fall in the intersection of SSL and CL. Fini et al. (2021) showed that self-supervised loss functions can be seamlessly converted into distillation losses for CL by adding a…”