A critical analysis of self-supervision, or what we can learn from a single image

By Yuki M. Asano, Christian Rupprecht, Andrea Vedaldi. 2019

153 citations recorded by the source corpus.

We look critically at popular self-supervision techniques for learning deep convolutional neural networks without manual labels. We show that three different and representative methods, BiGAN, RotNet and DeepCluster, can learn the first few layers of a convolutional network from a single image as well as using millions of images and manual labels, provided that strong data augmentation is used. However, for deeper layers the gap with manual supervision cannot be closed even if millions of unlabelled images are used for training. We conclude that: (1) the weights of the early layers of deep networks contain limited information about the statistics of natural images, that (2) such low-level statistics can be learned through self-supervision just as well as through strong supervision, and that (3) the low-level statistics can be captured via synthetic transformations instead of using a large image dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “…work has reported that data augmentation is useful for self-supervised learning (Doersch et al., 2015; Bachman et al., 2019; Hénaff et al., 2019; Asano et al., 2019), we show that data augmentation that does not yield accuracy beneﬁts for supervised learning can still help considerably with…”
Are Large-scale Datasets Necessary for Self-Supervised Pre-training? (2021, unknown): “Finally, some work studied how much could be learned from a single pre-training image [54] or from synthetic data [55, 56].”
Negative Data Augmentation (2021, methodology): “Data augmentation strategies for synthesizing new data in a way that is consistent with an underlying task are extremely effective in both supervised and unsupervised learning (Oord et al., 2018; Zhang et al., 2016; Noroozi & Favaro, 2016; Asano et al., 2019).”
Neighborhood Contrastive Learning for Novel Class Discovery (2021, methodology): “First, we learn a label-agnostic image representation by self-supervision learning [10] using both labeled and unlabeled datasets, which has been shown to be particularly good at extracting low-level features in the first layers of the network [2].”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, background): “Transformations of the input data are key to forcing the network to learn invariances in a self-supervised setting [66].”
Learning from One Continuous Video Stream (2023, background): “Relevant for our work are also attempts to train a ConvNet [22] or a ViT [9] from a single image [1] or a single long video [44].”
Test-Time Training with Self-Supervision for Generalization under Distribution Shifts (2019, background): “Asano et al. (2019) show that self-supervised learning on only a single image, surprisingly, can produce lowlevel features that generalize well.”
Test-Time Training for Out-of-Distribution Generalization (2019, background): “Particularly relevant to our work, Asano et al. (2019) shows that self-supervised learning on only a single image, surprisingly, can produce low-level features that generalize well.”
When Does Self-supervision Improve Few-shot Learning? (2019, background): “[2] showed that the representations can be learned with only one image and extreme augmentations.”
Unsupervised Single-Scene Semantic Segmentation for Earth Observation (2022, methodology): “The method focuses on single scene, thus further showing potential of deep self-supervised learning in data-constrained situation, similar to [50].”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “Asano et al. [14] demonstrated that SSL can produce generalizable features that exhibit robust generalization even when applied to a single image.”
Visualization of Supervised and Self-Supervised Neural Networks via Attribution Guided Factorization (2020, methodology): “4(b) shows the increase of segmentation performance for RotNet with our method, as we visualize deeper layers of SSL Alexnet, using a supervised linear post-training as proposed by Asano, Rupprecht, and Vedaldi (2019a).”
No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems (2020, unknown): “Techniques for learning semantically useful image features include autoencoder-based methods [32, 46], the use of unsupervised auxiliary tasks [2, 9], and pretraining on massive datasets [27].”
Hard Negative Mixing for Contrastive Learning (2020, unknown): “As revealed by recent studies [2, 17, 39, 43], heavy data augmentations applied to the same image are crucial in learning useful representations, as they modulate the hardness of the self-supervised task via the positive pair.”