What makes instance discrimination good for transfer learning?

By Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, Stephen Lin. 2020

183 citations recorded by the source corpus.

Unsupervised visual pretraining based on the instance discrimination pretext task has shown significant progress. Notably, in the recent work of MoCo, unsupervised pretraining has shown to surpass the supervised counterpart for finetuning downstream applications such as object detection on PASCAL VOC. It comes as a surprise that image annotations would be better left unused for transfer learning. In this work, we investigate the following problems: What makes instance discrimination pretraining good for transfer learning? What knowledge is actually learned and transferred from unsupervised pretraining? From this understanding of unsupervised pretraining, can we make supervised pretraining great again? Our findings are threefold. First, what truly matters for this detection transfer is low-level and mid-level representations, not high-level representations. Second, the intra-category invariance enforced by the traditional supervised model weakens transferability by increasing task misalignment. Finally, supervised pretraining can be strengthened by following an exemplar-based approach without explicit constraints among the instances within the same category.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, background): “Another possibility, explored in Zhao et al. (2021); Appalaraju et al. (2020); Ericsson et al. (2021), is to learn to invert the DN features through a Deep Image Prior (DIP) g θ as in min θ d ( g θ ( f ( x )) , x ) .”
How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (2021, methodology): “In Section 5.2, we adopt the CKA similarity (Kornblith et al., 2019a) and image reconstruction from features (Zhao et al., 2020) to directly analyze the representation.”
How Well Do Self-Supervised Models Transfer? (2020, methodology): “However, to complement our prior performance-driven comparisons, we conduct a preliminary analysis on this topic using the methodology suggested in [68].”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, background): “FSL leverages pre-trained models’ feature representation capacity to largely reduce intra-class variation [91], which leads to the model focusing on more discriminative regions.”
Masked Feature Prediction for Self-Supervised Visual Pre-Training (2021, methodology): “We mainly compare features from self-supervised models, which are considered to contain more diverse scene layout [9] and preserve more visual details [99] than features from supervised models.”
Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (2021, background): “One explanation is that self-supervised pre-training may keep more visual information, compared to supervised pre-training that mainly extracts information specific to classification [76].”
Hard Negative Mixing for Contrastive Learning (2020, background): “This is something that is further utilized in [86].”
Generative Hierarchical Features from Synthesizing Images (2020, background): “Thus the discriminative features learned from solving highlevel image classification tasks might not be necessarily good for other mid-level and low-level tasks, limiting their transferability [57, 64].”
Improving Visual Prompt Tuning for Self-supervised Vision Transformers (2023, background): “Depending on the pretraining strategy, the pretrained neural networks encode information differently in terms of the amount and content (Zhao et al., 2020; Bordes et al., 2021).”
Contrastive Representation Learning: A Framework and Review (2020, background): “Recent successes in transfer learning by instance discrimination contrastive pre-training [16], [43], [69] have raised the question of ‘‘what representation is learned from contrastive methods and why is it better than supervised pre-training’’ [101], [119]? However from the view of the Contrastive ”
Learn from Others and Be Yourself in Heterogeneous Federated Learning (2022, methodology): “For the heterogeneity problem, we take inspiration from the self-supervised learning [5, 6, 11, 13, 18, 25, 49, 91, 94].”
MixCo: Mix-up Contrastive Learning for Visual Representation (2020, background): “Such representations are highly transferable to down-stream tasks [5, 12, 19], and even outperform the representations learned by the supervised manner in some tasks [7, 35].”
Concept Generalization in Visual Representation Learning (2020, background): “Previous attempts discussing this issue have been limited to intuition [67, 75].”
Dense Contrastive Learning for Self-Supervised Visual Pre-Training (2020, background): “Generally speaking, the success of self-supervised learning [41, 17, 42, 47, 16, 14] can be attributed to two important aspects namely contrastive learning, and pretext tasks.”
Debiased Pseudo Labeling in Self-Training (2022, background): “One possible reason is that the representations learned by different pre-trained models focus on different aspects of the data (Zhao et al., 2021).”
How Well Self-Supervised Pre-Training Performs with Streaming Data? (2021, background): “In addition, features learned by the self-supervised task of instance discrimination are able to keep more visual information than the features learned by supervised pre-training [64], which weakens the effect of knowledge forgetting during sequential training.”