Scaling and Benchmarking Self-Supervised Visual Representation Learning

By Priya Goyal, Dhruv Kumar Mahajan, Abhinav Kumar Gupta, Ishan Misra. 2019

432 citations recorded by the source corpus.

Self-supervised learning aims to learn representations from the data itself without explicit manual supervision. Existing efforts ignore a crucial aspect of self-supervised learning - the ability to scale to large amount of data because self-supervision requires no manual labels. In this work, we revisit this principle and scale two popular self-supervised approaches to 100 million images. We show that by scaling on various axes (including data size and problem 'hardness'), one can largely match or even exceed the performance of supervised pre-training on a variety of tasks such as object detection, surface normal estimation (3D) and visual navigation using reinforcement learning. Scaling these methods also provides many interesting insights into the limitations of current self-supervised techniques and evaluations. We conclude that current self-supervised methods are not 'hard' enough to take full advantage of large scale data and do not seem to learn effective high level semantic representations. We also introduce an extensive benchmark across 9 different datasets and tasks. We believe that such a benchmark along with comparable evaluation settings is necessary to make meaningful progress. Code is at: https://github.com/facebookresearch/fair_self_supervision_benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Scaling & Learning Theory

Citations in the corpus

Momentum Contrast for Unsupervised Visual Representation Learning (2019, methodology): “The Jigsaw [45] result is from the ResNet-based implementation in [26].”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “However, self-supervised models are now performing well on tasks such as surface normal estimation, detection, and navigation [19].”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, methodology): “The original paper (Goyal et al., 2019a) also includes navigation and surface normal estimation.”
Learning Representations by Maximizing Mutual Information Across Views (2019, background): “Some potential shortcomings with standard evaluation protocols have been noted by Goyal et al. [2019].”
Learning Visual Representations with Caption Annotations (2020, methodology): “Following the convention of [6], we train linear SVMs to evaluate visual representations on the 2007 split of Pascal-VOC and the 2017 split of MS-COCO datasets, in a one-vs.”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, background): “There are other approaches to self-supervised learning that are based on handcrafted pretext tasks [52, 31, 53, 54, 27, 55].”
VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “Some recent approaches have begun to match or exceed supervised pretraining on ImageNet [22–26], and have been scaled to hundreds of millions [22, 25, 27, 28] or billions [24] of images.”
Self-supervised Pretraining of Visual Features in the Wild (2021, background): “Prior attempts to train self-supervised models on uncurated data [6, 13, 19, 27] have only used a few millions of images for which using small model architectures is sufﬁcient.”
SLIP: Self-supervision meets Language-Image Pre-training (2021, methodology): “Prior work on representation learning has argued against end-to-end ﬁnetuning for its sensitivity to optimization hyperparameters [24], and against linear classiﬁcation for being too contrived [63].”
Self-Supervised Learning of Pretext-Invariant Representations (2019, methodology): “Prior work suggests that the hyperparameters used in transfer learning can play an important role in the evaluation of pre-trained representations [21, 35, 87].”
A critical analysis of self-supervision, or what we can learn from a single image (2019, neutral): “Our results complement recent works (Mahajan et al., 2018; Goyal et al., 2019) that have investigated self-supervision in the very large data regime.”
Unified Contrastive Learning in Image-Text-Label Space (2022, background): “Self-supervised learning (SSL) for vision aims to learn general-purpose visual representations from raw pixels without supervisions from label or text [19].”
Prototypical Contrastive Learning of Unsupervised Representations (2020, positive): “We follow previous setup [37, 4] and evaluate the performance of such linear classifiers on three datasets, including ImageNet, VOC07, and Places205.”
The Challenges of Continuous Self-Supervised Learning (2022, methodology): “Even methods trained on larger datasets like Instagram-1B [ 24 , 26 ] are less likely to encounter heavily correlated samples in the mini-batches.”
Barlow Twins: Self-Supervised Learning via Redundancy Reduction (2021, methodology): “We follow standard practice (Goyal et al., 2019) and evaluate our representations by transfer learning to different datasets and tasks in computer vision.”
SUPERB: Speech processing Universal PERformance Benchmark (2021, background): “Well-established benchmark, such as GLUE [5] in NLP and VISSL [6] in CV, is essential to evaluate pretrained models’ generalizability and re-usability across a wide range of tasks.”
Few-Shot Learning via Learning the Representation, Provably (2020, methodology): “In particular, representation learning is widely adopted for few-shot learning tasks (Sun et al., 2017, Goyal et al., 2019, Arora et al., 2020).”
TextCaps: a Dataset for Image Captioning with Reading Comprehension (2020, background): “In recent years, with the availability of large labelled corpora, progress in image captioning has seen steady increase in performance and quality [4,10,12,13,34] and reading scene text (OCR) has matured [8,16,19,21,31].”
Audio-Visual Instance Discrimination with Cross-Modal Agreement (2020, methodology): “We confirmed this observation by transferring both Cross-AVID and AVID+CMA representations learned on the full Audioset dataset to the action recognition task on the Kinetics dataset by linear probing [22, 88].”
Robust fine-tuning of zero-shot models (2021, background): “, image classification) is an effective pre-training objective [89, 51, 25, 102], more scalable methods that do not require labeled data have attracted significant interest [24, 74, 75, 33, 35, 15, 13, 14, 69, 12].”