Emerging Properties in Self-Supervised Vision Transformers
By Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J'egou, Julien Mairal, Piotr Bojanowski, Armand Joulin. IEEE International Conference on Computer Vision, 2021
9,706 citations recorded by the source corpus.
In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) [16] that stand out compared to convolutional networks (convnets). Beyond the fact that adapting self-supervised methods to this architecture works particularly well, we make the following observations: first, self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs, nor with convnets. Second, these features are also excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT. Our study also underlines the importance of momentum encoder [26], multi-crop training [9], and the use of small patches with ViTs. We implement our findings into a simple self-supervised method, called DINO, which we interpret as a form of self-distillation with no labels. We show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning
View Paper: https://arxiv.org/abs/2104.14294
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Yoshua Bengio
- Quoc V. Le
- Trevor Darrell
- Sergey Levine
- Dacheng Tao
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Andrea Vedaldi
In Representation & Self-Supervised Learning
- Andrea Vedaldi
- Ishan Misra
- Andrew Zisserman
- Trevor Darrell
- Armand Joulin
- Kaiming He
- Yann LeCun
- Alexei A. Efros
Citations in the corpus
- Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “In particular, computer vision models trained with the latest contrastive learning methods led to features well suited to retrieval (Caron et al., 2021).”
- Benchmarking Detection Transfer Learning with Vision Transformers (2021, methodology): “…role in the study of transfer learning over the last decade ( e.g ., [35, 14, 9, 17]), was not explored in the pioneering work on ViT training [10, 7, 5]—supervised or unsupervised—due to the challenges (described shortly) of integrating ViTs into common detection models, like Mask R-CNN [19].”
- BEiT: BERT Pre-Training of Image Transformers (2021, methodology): “Moreover, we compare BEIT with previous state-of-the-art self-supervised methods for Transformer, such as DINO (Caron et al., 2021), and MoCo v3 (Chen et al.”
- An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “The gain of using small patches is also observed in [8].”
- Vector-quantized Image Modeling with Improved VQGAN (2021, methodology): “In computer vision, in contrast, most recent unsupervised or self-supervised learning research focuses on applying different random augmentations to images, with the pretraining objective to distinguish image instances (Chen et al., 2020b; He et al., 2020; Chen et al., 2020d; Grill et al., 2020; Che”
- SLIP: Self-supervision meets Language-Image Pre-training (2021, methodology): “Recent methods for self-supervised learning also propose a variety of alternatives to the contrastive objective such as self-distillation [25, 6], or input reconstruction [1, 26].”
- High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, positive): “…modern family of SSL approaches for images (Misra & Maaten, 2020; Chen et al., 2020; Chen & He, 2020; He et al., 2020; Grill et al., 2020; Caron et al., 2020; 2021; Zbontar et al., 2021; Bardes et al., 2021), have two noteworthy characteristics that markedly distinguish them from…”
- XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, background): “2019), computer vision (Chen et al., 2020; He et al., 2020; Caron et al., 2021) as well as speech processing (van den Oord et al.”
- LiT: Zero-Shot Transfer with Locked-image text Tuning (2021, background): “Fu ll IN 10 -s ho t 0- sh ot I→ T T → I MoCo-v3 [10] IN n 76.7 60.6 55.4 33.5 17.6 DINO [4] IN n 78.2 61.2 55.5 33.4 18.2 AugReg [54] IN21k y 77.4 63.9 55.9 30.3 17.2 AugReg [54] IN y 77.7 77.1 64.3 25.4 13.8 AugReg [54] Places y - 22.5 28.5 25.1 12.9 Table 3 shows model performance without LiT (Ima”
- Scaling Vision Transformers (2021, methodology): “In this paper we concentrate on scaling laws for transfer performance of ViT models pre-trained on image classifica- ?equal contribution 1 105 25 Examples per class 60 65 70 75 80 85 Im ag eN et to p- 1 ac cu ra cy Our ViT-G ViT-H SimCLR v2 BYOL DINO Figure 1.”
- Unified Contrastive Learning in Image-Text-Label Space (2022, background): “This augmented-view-based paradigm has also been extended to non-contrastive methods [4,7,20,37], where only positive image view pairs are considered in learning.”
- MaskGIT: Masked Generative Image Transformer (2022, background): “The transformer architecture [46], was first proposed in NLP, and has recently extended its reach to computer vision [6, 13].”
- How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers (2021, background): “Alternatively, [4] devises a self-supervised training procedure based on the idea from [12], achieving impressive results.”
- Unsolved Problems in ML Safety (2021, background): “They frequently demonstrate properties of self-organizing systems such as spontaneously emergent capabilities [23, 32].”
- Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replay (2022, background): “Besides, inspired by the remarkable progress in self-supervised visual representation learning [1,2,6,7,19], [12] propose to enable continual learning without labeled data.”
- Class-incremental Novel Class Discovery (2022, background): “In an attempt to realize this, much effort has been dedicated to learn deep learning models from large reservoirs of both labelled [24,17,10] and unlabelled data [3,4].”