Continual Unsupervised Representation Learning

By Dushyant Rao, Francesco Visin, Andrei A. Rusu, Yee Whye Teh, Razvan Pascanu, Raia Hadsell. 2019

293 citations recorded by the source corpus.

Continual learning aims to improve the ability of modern learning systems to deal with non-stationary distributions, typically by attempting to learn a series of tasks sequentially. Prior art in the field has largely considered supervised or reinforcement learning tasks, and often assumes full knowledge of task labels and boundaries. In this work, we propose an approach (CURL) to tackle a more general problem that we will refer to as unsupervised continual learning. The focus is on learning representations without any knowledge about task identity, and we explore scenarios when there are abrupt changes between tasks, smooth transitions from one task to another, or even when the data is shuffled. The proposed approach performs task inference directly within the model, is able to dynamically expand to capture new concepts over its lifetime, and incorporates additional rehearsal-based techniques to deal with catastrophic forgetting. We demonstrate the efficacy of CURL in an unsupervised learning setting with MNIST and Omniglot, where the lack of labels ensures no information is leaked about the task. Further, we demonstrate strong performance compared to prior art in an i.i.d setting, or when adapting the technique to supervised tasks such as incremental class learning.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Radiology, Nuclear Medicine and Imaging

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “According to the provided X t and Y t in each D t , continual learning is further extended to zero-shot [23], few-shot [24], semi-supervised [25], open-world [26] and un-/self-supervised [27], [28] scenarios.”
Progressive Prompts: Continual Learning for Language Models (2023, methodology): “Following previous studies (Rao et al., 2019; de Masson D’Autume et al., 2019), for CL experiments, for each dataset we use the available validation set as a test set (since test data is not available), and hold out 500 samples from the train set to construct the validation set.”
A soft nearest-neighbor framework for continual semi-supervised learning (2022, neutral): “While the vast majority of these methods operate in a supervised setting, recent works addressed the problem of overcoming catastrophic forgetting in the challenging case of limited or no supervision [1, 8, 21, 28, 38, 46, 47, 53].”
Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, background): “CURL [36] proposes a continual unsupervised representation learning method, which learns a task-specific representation based on a set of share parameters and also trains a generative model to avoid forgetting.”
Task-Free Continual Learning via Online Discrepancy Distance Learning (2022, neutral): “Another kind of approaches for continual learning focuses on the memory system, which usually employs a small memory buffer [1, 5, 6, 28, 36, 44, 59] to store a few past data or trains a generator to provide the replay samples when learning new tasks [38, 43, 46, 47, 52, 58, 53].”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, neutral): “Replayed-based methods leverage episodic memory to store representative history data or generate virtual data via a generative model, and replay these samples with current data [2, 7, 8, 12, 26, 37, 39, 40, 45, 50, 51].”
The Challenges of Continuous Self-Supervised Learning (2022, background): “However, recent works [1, 63, 66, 82] have attempted to address a sub-problem of ours, i.”
Learning to Prompt for Continual Learning (2021, background): “The task-specific components can be identified by expanding the network [26, 31, 48, 50, 64, 68], or attend to task-specific sub-networks [19,35,51,59].”
Self-Supervised Models are Continual Learners (2021, methodology): “Finally, we do not consider methods based on VAEs [1,45], since they have been shown to yield poor performance on large scale.”
Representational Continuity for Unsupervised Continual Learning (2021, background): “Although Rao et al. (2019) instantiated a continual unsupervised representation learning framework (C URL ), it is not scalable for high-resolution tasks, as it is composed of MLP encoders/decoders and a simple MoG generative replay.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “A few other methods [212, 213] explore the combination of generative replay and parameter isolation based methods for image classification.”
Continual World: A Robotic Benchmark For Continual Reinforcement Learning (2021, background): “[39] Dushyant Rao, Francesco Visin, Andrei A.”
How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (2021, background): “Although works (Rao et al., 2019; Aljundi et al., 2019) explore continual learning for some speciﬁc unsupervised tasks, few have studied the transfer learning performance of sequential self-supervised models.”
Online Continual Learning in Image Classification: An Empirical Survey (2021, background): “Although CL is also studied in reinforcement learning [19, 20, 21] and more recently in unsupervised learning [22], the image classification problem is still the main focus for many CL researchers.”
ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning (2021, background): “However, FSIL and UCL only make use of a small amount of labeled data or large amounts of unlabeled data, both of which are easy to acquire in many real-world applications.”
Embracing Change: Continual Learning in Deep Neural Networks. (2020, methodology): “Rather, generative models are trained and then used to generate rehearsal data as needed [69 – 72].”
Linear Mode Connectivity in Multitask and Continual Learning (2020, methodology): “Parameter isolation: Parameter isolation methods allocate different subsets of the parameters to each task (Rusu et al., 2016; Yoon et al., 2018; Jerfel et al., 2019; Rao et al., 2019; Li et al., 2019).”
Continual Prototype Evolution: Learning Online from Non-Stationary Data Streams (2020, methodology): “From the expansion-based methods DN-CPM is competitive, whereas CURL is more suited for unsupervised learning and lags behind.”
Learning latent representations across multiple data domains using Lifelong VAEGAN (2020, background): “in a broad range of tasks. Learning meaningful and disentangled representations of data was shown to benet many tasks, but they have not been explored so far within the lifelong learning methodology [1,52]. In this paper, we propose a new lifelong learning model which not only learns a GRM but also ”