Revisiting Self-Supervised Visual Representation Learning

By Alexander Kolesnikov, Xiaohua Zhai, Lucas Beyer. 2019

767 citations recorded by the source corpus.

Unsupervised visual representation learning remains a largely unsolved problem in computer vision research. Among a big body of recently proposed approaches for unsupervised learning of visual representations, a class of self-supervised techniques achieves superior performance on many challenging benchmarks. A large number of the pretext tasks for self-supervised learning have been studied, but other important aspects, such as the choice of convolutional neural networks (CNN), has not received equal attention. Therefore, we revisit numerous previously proposed self-supervised models, conduct a thorough large scale study and, as a result, uncover multiple crucial insights. We challenge a number of common practices in self-supervised visual representation learning and observe that standard recipes for CNN design do not always translate to self-supervised representation learning. As part of our study, we drastically boost the performance of previously proposed techniques and outperform previously published state-of-the-art results by a large margin. We will release the code for reproducing our experiments when the anonymity requirements are lifted.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “Table 6 compares our results with previous approaches (Zhuang et al., 2019; He et al., 2019a; Misra & van der Maaten, 2019; Hénaff et al., 2019; Kolesnikov et al., 2019; Donahue & Simonyan, 2019; Bachman et al., 2019; Tian et al., 2019) in the linear evaluation setting.”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “Following [12, 33], to scale this approach to hundreds of millions of images (frames), we employ a triplet loss [52] encouraging augmentations of the same image to be close and augmentations of different images to be far apart.”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “We ﬁnd that bigger architectures perform better on VTAB (Appendix L and Kolesnikov et al. (2019)).”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, methodology): “Recent work has shown that larger networks and more effective training improves self-supervised learning (Doersch & Zisserman, 2017; Kolesnikov et al., 2019), but the original CPC model used only the first 3 stacks of a ResNet-101 architecture.”
S4L: Self-Supervised Semi-Supervised Learning (2019, methodology): “We closely follow the protocol defined by [16].”
Learning Visual Representations with Caption Annotations (2020, neutral): “see [33]).”
Self-Supervised MultiModal Versatile Networks (2020, methodology): “For video we explore using S3DG [90] (dv = 1024), and TSM [47] with a ResNet50 backbone (dv = 2048) or a ResNet50x2 backbone (ResNet50 with all channels doubled [42], dv = 4096).”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, methodology): “There are other approaches to self-supervised learning that are based on handcrafted pretext tasks [52, 31, 53, 54, 27, 55].”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “Linear evaluation on ImageNet We first evaluate BYOL’s representation by training a linear classifier on top of the frozen representation, following the procedure described in [48, 74, 41, 10, 8], and appendix D.”
Contrastive Multiview Coding (2019, neutral): “On top of it, strengthening data augmentation with RandAugment [14] yields better or comparable results to other state-of-the-art methods [41, 79, 87, 27, 49, 19, 29, 4].”
Self-supervised Pretraining of Visual Features in the Wild (2021, background): “[30] have demonstrated the importance of training wider networks for the quality of visual features learned with self-supervision.”
Self-Supervised Learning of Pretext-Invariant Representations (2019, methodology): “Prior work suggests that the hyperparameters used in transfer learning can play an important role in the evaluation of pre-trained representations [21, 35, 87].”
On Mutual Information Maximization for Representation Learning (2019, methodology): “Following the widely adopted downstream linear evaluation protocol (Kolesnikov et al., 2019; van den Oord et al., 2018; Hénaff et al., 2019; Tian et al., 2019; Hjelm et al., 2019; Bachman et al., 2019), we train a linear classiﬁer 3 for digit classiﬁcation on the learned representation using all…”
Large Scale Adversarial Representation Learning (2019, methodology): “6 Rotation [9, 20] RevNet-50 ×4 AvePool 55.”
Self-Supervised GANs via Auxiliary Rotation Loss (2018, background): “Recently, Kolesnikov et al. [46] conducted a study on self-supervised learning with modern neural architectures.”
ScaleNet: An Unsupervised Representation Learning Method for Limited Information (2023, background): “RotNet is one of the simplest self-supervised learning approaches that capture visual representation via rotation-prediction task [28].”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, methodology): “While recent approaches [14, 33, 72] used models like ResNet50 or 101, we explore the relationship between model capacity and data size which we believe is crucial for future efforts in self-supervised learning.”
CrossTransformers: spatially-aware few-shot transfer (2020, background): “Making our networks spatially-aware requires higher-resolution, and also higher-capacity networks are especially important in self-supervised learning [23, 45].”