An Empirical Study of Training Self-Supervised Vision Transformers

By Xinlei Chen, Saining Xie, Kaiming He. 2021

2,433 citations recorded by the source corpus.

This paper does not describe a novel method. Instead, it studies a straightforward, incremental, yet must-know baseline given the recent progress in computer vision: self-supervised learning for Vision Transformers (ViT). While the training recipes for standard convolutional networks have been highly mature and robust, the recipes for ViT are yet to be built, especially in the self-supervised scenarios where training becomes more challenging. In this work, we go back to basics and investigate the effects of several fundamental components for training self-supervised ViT. We observe that instability is a major issue that degrades accuracy, and it can be hidden by apparently good results. We reveal that these results are indeed partial failure, and they can be improved when training is made more stable. We benchmark ViT results in MoCo v3 and several other self-supervised frameworks, with ablations in various aspects. We discuss the currently positive evidence as well as challenges and open questions. We hope that this work will provide useful data points and experience for future research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Representation & Self-Supervised Learning

Citations in the corpus

Masked Autoencoders Are Scalable Vision Learners (2021, methodology): “, MoCo v3 [9] saturates at 300 epochs for ViT-L.”
Benchmarking Detection Transfer Learning with Vision Transformers (2021, methodology): “…role in the study of transfer learning over the last decade ( e.g ., [35, 14, 9, 17]), was not explored in the pioneering work on ViT training [10, 7, 5]—supervised or unsupervised—due to the challenges (described shortly) of integrating ViTs into common detection models, like Mask R-CNN [19].”
Early Convolutions Help Transformers See Better (2021, background): “[4] who report ViT optimization instability in self-supervised learning (unlike with CNNs), and find that freezing the patchify stem at its random initialization improves stability.”
BEiT: BERT Pre-Training of Image Transformers (2021, methodology): “Apart from masked auto-encoding, other mainstream research works use contrastive learning (Chen et al., 2021; Xie et al., 2021), and self-distillation (Caron et al.”
Exploring Simple Siamese Representation Learning (2020, methodology): “, Emerging Properties in Self-Supervised Vision Transformers, 2021 [2] Chen et al.”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “In particular, contrastive representation learning encourages representations to be closer between “positive” pairs and further between “negative” pairs; contrastive predictive coding (CPC, [95]), based on multi-class classification, have been commonly used in state-of-the-art methods [37, 16, 18, 1”
CogView: Mastering Text-to-Image Generation via Transformers (2021, positive): “It seems to slow down the updating of token embeddings, but actually does not hurt performance in our experiments, which also corresponds to a recent work MoCo v3 [9].”
SLIP: Self-supervision meets Language-Image Pre-training (2021, methodology): “The memory bank kNN monitor [11] alleviates this cost but is not feasible when pre-training on unlabeled datasets such as YFCC100M.”
Stable and low-precision training for large-scale vision-language models (2023, methodology): “Loss spikes can emerge when scaling up models [8, 24, 13, 67, 69, 52, 72].”
GLM-130B: An Open Bilingual Pre-trained Model (2022, neutral): “The problem is handled in vision models (Chen et al., 2021) via freezing the patch projection layer.”
LiT: Zero-Shot Transfer with Locked-image text Tuning (2021, background): “Fu ll IN 10 -s ho t 0- sh ot I→ T T → I MoCo-v3 [10] IN n 76.7 60.6 55.4 33.5 17.6 DINO [4] IN n 78.2 61.2 55.5 33.4 18.2 AugReg [54] IN21k y 77.4 63.9 55.9 30.3 17.2 AugReg [54] IN y 77.7 77.1 64.3 25.4 13.8 AugReg [54] Places y - 22.5 28.5 25.1 12.9 Table 3 shows model performance without LiT (Ima”
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, methodology): “Checkpoints Taxonomy Pre-training Settings Network Architecture Language Knowledge Training Objective Dataset Vision Language Others Image Classification MoCo-v3 [10] 7 7 Self-Supervised ImageNet-1K (1.”
Unified Contrastive Learning in Image-Text-Label Space (2022, background): “Contrastive learning has laid the foundation for the best performing SSL models [3, 6, 8, 21, 24, 55, 56].”
Visual Prompt Tuning (2022, methodology): “Backbone Pre-trained Objective Pre-trained Dataset # params (M) Feature dim d Batch Size Pre-trained Model ViT-B/16 [16] Supervised ImageNet-21k 85 768 2048 / 1280 / 128 / 128 / 64 checkpoint ViT-L/16 [16] 307 1024 2048 / 640 / 64 / 64 / 32 checkpoint ViT-H/14 [16] 630 1280 1024 / 240 / 28 / 28 / 14”
SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model (2023, methodology): “Meanwhile, the use of SL allows MoCo v3 [4] to learn representations much closer to that of the joint training (Fig.”
The Challenges of Continuous Self-Supervised Learning (2022, background): “After relentless improvements in image augmentations [12, 46], backbone models [10, 24], stable (slow-moving) learning targets [9, 15, 29], and transformation invariant loss functions [10, 13, 27, 56, 84], augmentation invariance has produced impressive models that improve state-of-the-art on a dive”
Transformers in Vision: A Survey (2021, methodology): “[115] evaluate different self-supervised frameworks and propose practical strategies including MoCo v3 (extended from v1/v2 [116], [117]) for stabilized training of self-supervised ViTs.”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “[20] conduct contrastive learning using ViT [29] and achieve impressive results.”
ImageNet-21K Pretraining for the Masses (2021, background): “Previous works have observed that MLP-based models can be harder and less stable to use in transfer learning since they don’t have inherent translation inductive bias [6, 39, 35].”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, methodology): “These advancements further inspired the applications of transformer to various vision tasks beyond image classification, such as self-supervised learning [16, 10, 40], object detection [9, 91, 84, 20] and semantic segmentation [61, 65, 86].”