How Well Do Self-Supervised Models Transfer?

By Linus Ericsson, Henry Gouk, Timothy M. Hospedales. 2020

327 citations recorded by the source corpus.

Self-supervised visual representation learning has seen huge progress recently, but no large scale evaluation has compared the many models now available. We evaluate the transfer performance of 13 top self-supervised models on 40 downstream tasks, including many-shot and few-shot recognition, object detection, and dense prediction. We compare their performance to a supervised baseline and show that on most tasks the best self-supervised models outperform supervision, confirming the recently observed trend in the literature. We find ImageNet Top-1 accuracy to be highly correlated with transfer to many-shot recognition, but increasingly less so for few-shot, object detection and dense prediction. No single self-supervised method dominates overall, suggesting that universal pre-training is still unsolved. Our analysis of features suggests that top self-supervised learners fail to preserve colour information as well as supervised alternatives, but tend to induce better classifier calibration, and less attentive overfitting than supervised learners.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, neutral): “Another possibility, explored in Zhao et al. (2021); Appalaraju et al. (2020); Ericsson et al. (2021), is to learn to invert the DN features through a Deep Image Prior (DIP) g θ as in min θ d ( g θ ( f ( x )) , x ) .”
How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (2021, methodology): “Both classiﬁcation protocols are the same as (Ericsson et al., 2021).”
Self-Supervised Training Enhances Online Continual Learning (2021, background): “In contrast, self-supervised learning promotes the acquisition of category agnostic feature representations [9, 22].”
Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (2021, methodology): “Following [13], we use ResNet-50 (1 × ), pre-trained by various CSL methods on ImageNet, as the network backbone.”
Concept Generalization in Visual Representation Learning (2020, unknown): “When it comes to evaluating the quality of visual representations, the gold standard is to benchmark models by solving diverse tasks such as classification, detection, segmentation and retrieval on many datasets [4, 7, 13, 18, 22, 29, 73].”
Improving CLIP Training with Language Rewrites (2023, methodology): “Few-shot Classification Following the settings in [18], we evaluate the 5-way 5-shot performance across 15 downstream datasets.”
How Well Self-Supervised Pre-Training Performs with Streaming Data? (2021, methodology): “Following [13], the transfer performance of pre-trained models are evaluated on 12 image classiﬁcation datasets [24] and the Pascal VOC [14] detection.”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “The advancement of SSL [3], [4], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24] has exhibitedrapidprogress, capturing signiﬁcant attention within the research community (Fig.”
A Cookbook of Self-Supervised Learning (2023, unknown): “…learning algorithms, express seemingly contradictory findings, namely that existing self-supervised learning strategies are or are not effective for downstream dense prediction tasks [Goyal et al., 2019, Purushwalkam and Gupta, 2020, Zhao et al., 2021, Ericsson et al., 2021b, Shwartz-Ziv et al.].”
Self-Supervised Representation Learning: Introduction, advances, and challenges (2021, unknown): “There have been mixed results reported in the literature with regard to whether linear readout is sufficient or whether fine-tuning the entire encoder should improve performance [29], [30].”