Self-Supervised Representation Learning: Introduction, advances, and challenges

By Linus Ericsson, Henry Gouk, Chen Change Loy, Timothy M. Hospedales. 2021

410 citations recorded by the source corpus.

Self-supervised representation learning (SSRL) methods aim to provide powerful, deep feature learning without the requirement of large annotated data sets, thus alleviating the annotation bottleneck—one of the main barriers to the practical deployment of deep learning today. These techniques have advanced rapidly in recent years, with their efficacy approaching and sometimes surpassing fully supervised pretraining alternatives across a variety of data modalities, including image, video, sound, text, and graphs. This article introduces this vibrant area, including key concepts, the four main families of approaches and associated state-of-the-art techniques, and how self-supervised methods are applied to diverse modalities of data. We further discuss practical considerations including workflows, representation transferability, and computational cost. Finally, we survey major open challenges in the field, that provide fertile ground for future work.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Continual Barlow Twins: Continual Self-Supervised Learning for Remote Sensing Semantic Segmentation (2022, background): “Recently, referring to CV in general, several innovations have been proposed [53], [54], reviewed in many surveys [24], [55], [56], [57], [58].”
Self-Supervised Speech Representation Learning: A Review (2022, background): “Due to the popularity of SSL, reviews have been published about the technology in general [16], [17], [18] as well as its application to natural language processing (NLP) [19], [20],”
SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation (2022, background): “This is even enforced by minimizing similarities between anchor and different images in a batch as they have different global structures [8,17].”
BockNet: Blind-Block Reconstruction Network With a Guard Window for Hyperspectral Anomaly Detection (2023, background): “Self-SL comprises two key steps: 1) constructing input samples and corresponding supervision information on unlabeled data and 2) designing the supporting pretext task according to specific target tasks, aiming to extract the intrinsic data features that are profitable to the target task [59].”