How Useful Is Self-Supervised Pretraining for Visual Tasks?

By Alejandro Newell, Jia Deng. 2020

151 citations recorded by the source corpus.

Recent advances have spurred incredible progress in self-supervised pretraining for vision. We investigate what factors may play a role in the utility of these pretraining methods for practitioners. To do this, we evaluate various self-supervised algorithms across a comprehensive array of synthetic datasets and downstream tasks. We prepare a suite of synthetic data that enables an endless supply of annotated images as well as full control over dataset difficulty. Our experiments offer insights into how the utility of self-supervision changes as the number of available labels grows as well as how the utility changes as a function of the downstream task and the properties of the training data. We also find that linear evaluation does not correlate with finetuning performance. Code and data is available at \href{https://www.github.com/princeton-vl/selfstudy}{github.com/princeton-vl/selfstudy}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (2021, background): “These phenomena are due to the so-called “negative transfer” (Wang et al., 2019), which is also discussed in other model pre-training studies (Newell & Deng, 2020; Gururangan et al., 2020).”
Self-Supervised Learning: Generative or Contrastive (2020, background): “Some following work [84], [153] investigates this problem and attributes it to the gap between the instance discrimination and object detection.”
Self-Supervised Remote Sensing Feature Learning: Learning Paradigms, Challenges, and Future Works (2022, background): “[34], [35] and computer vision (CV) [36], [37], [38].”
Geographical Knowledge-Driven Representation Learning for Remote Sensing Images (2021, background): “Each class is represented by a value among [10, 20, 30, 40, 50, 60, 70, 80, 90, 100].”
Self-Supervised Pretraining Improves Self-Supervised Pretraining (2021, methodology): “Here, we investigate the robustness of HPT to common factors that impact the effectiveness of self-supervised pretraining such as the augmentation policy [5, 48] and pretraining dataset size [36].”
Exploring Balanced Feature Spaces for Representation Learning (2021, methodology): “Since a pre-trained representation model is often used to facilitate downstream tasks (He et al., 2020; Newell & Deng, 2020; H´enaff et al., 2019), we here conduct extensive experiments to study its potential beneﬁts on model generalization performance under the following settings.”
GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning (2021, methodology): “Self-supervised Auxiliary Task Self-supervised pretraining has gradually emerged (Doersch and Zisserman, 2017; Newell and Deng, 2020) as a effective technique to deal with label scarcity and improve model performance.”