The Challenges of Continuous Self-Supervised Learning

By Senthil Purushwalkam, Pedro Morgado. 2022

59 citations recorded by the source corpus.

Self-supervised learning (SSL) aims to eliminate one of the major bottlenecks in representation learning - the need for human annotations. As a result, SSL holds the promise to learn representations from data in-the-wild, i.e., without the need for finite and static datasets. Instead, true SSL algorithms should be able to exploit the continuous stream of data being generated on the internet or by agents exploring their environments. But do traditional self-supervised learning approaches work in this setup? In this work, we investigate this question by conducting experiments on the continuous self-supervised learning problem. While learning in the wild, we expect to see a continuous (infinite) non-IID data stream that follows a non-stationary distribution of visual concepts. The goal is to learn a representation that can be robust, adaptive yet not forgetful of concepts seen in the past. We show that a direct application of current methods to such continuous setup is 1) inefficient both computationally and in the amount of data required, 2) leads to inferior representations due to temporal correlations (non-IID data) in some sources of streaming data and 3) exhibits signs of catastrophic forgetting when trained on sources with non-stationary data distributions. We propose the use of replay buffers as an approach to alleviate the issues of inefficiency and temporal correlations. We further propose a novel method to enhance the replay buffer by maintaining the least redundant samples. Minimum redundancy (MinRed) buffers allow us to learn effective representations even in the most challenging streaming scenarios composed of sequential visual data obtained from a single embodied agent, and alleviates the problem of catastrophic forgetting when learning from data with non-stationary semantic distributions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “MinRed [194] further promotes the diversity of experience replay by de-correlating the stored old training samples.”
Towards Label-Efficient Incremental Learning: A Survey (2023, background): “Buffer Self-Supervised Learning [Purushwalkam et al., 2022] is a memory-based approach that extends an off-the-shelf self-supervision algorithm, SimSiam [Chen and He, 2021] for the case of incremental learning.”
Continual Barlow Twins: Continual Self-Supervised Learning for Remote Sensing Semantic Segmentation (2022, background): “In CV, the combination of these two technologies is starting to be explored, and some works have already started to demonstrate how SSL methods are feasible to learn incrementally [17], [29], [30].”
Learning from One Continuous Video Stream (2023, background): “An inspiring but rare exception [34] investigated broadly the challenges of self-supervised learning from continuous data streams and proposed using a minimum-redundancy “replay” buffer [28] to deal with temporal correlations and learn more robust representations.”