Training data-efficient image transformers & distillation through attention
By Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, Herv'e J'egou. 2020
9,450 citations recorded by the source corpus.
Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. However, these visual transformers are pre-trained with hundreds of millions of images using an expensive infrastructure, thereby limiting their adoption. In this work, we produce a competitive convolution-free transformer by training on Imagenet only. We train them on a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop evaluation) on ImageNet with no external data. More importantly, we introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention. We show the interest of this token-based distillation, especially when using a convnet as a teacher. This leads us to report results competitive with convnets for both Imagenet (where we obtain up to 85.2% accuracy) and when transferring to other tasks. We share our code and models.
Subfields: Computer Vision and Pattern Recognition, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Yi Yang
- Alan L. Yuille
In Efficiency & Compression
Citations in the corpus
- Natural Adversarial Examples (2019, methodology): “We evaluate with DeiT [50], a vision Transformer trained on ImageNet-1K with aggressive data augmentation such as Mixup.”
- Benchmarking Detection Transfer Learning with Vision Transformers (2021, methodology): “We use the DeiT released weights [36] for ViT-B and the ViT-L weights from [16], which uses an even stronger training formula than DeiT to avoid overfitting (moreover, the DeiT release does not include ViT-L).”
- Early Convolutions Help Transformers See Better (2021, methodology): “We use a simplified training recipe compared to recent work such as DeiT [41], which we found to be equally effective across a wide spectrum of model complexities and dataset scales.”
- VOLO: Vision Outlooker for Visual Recognition (2021, methodology): “DeiT [50] and T2T-ViT [66] mitigate the problem of ViTs requiring large-scale datasets and propose data efficient ViTs.”
- Towards Robust Vision Transformer (2021, methodology): “Base on this seminal work, numerous variants have been proposed to improve ViTs from different perspectives containing training data efficiency [40], self-attention mechanism [25], introducing convolution [23,45,50] or pooling layers [20,43], etc.”
- Emerging Properties in Self-Supervised Vision Transformers (2021, methodology): “Transformers [70] have recently emerged as an alternative to convolutional neural networks (convnets) for visual recognition [19, 69, 83].”
- An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “By default, we use AdamW [30] as the optimizer, which is the common choice for training ViT models [15, 41, 35].”
- Projected GANs Converge Faster (2021, methodology): “Lastly, we utilize a vision transformer architecture (ViTBase) [9] and its efficient follow-up (DeiT-small distilled) [62].”
- SLIP: Self-supervision meets Language-Image Pre-training (2021, methodology): “For finetuning on smaller downstream datasets, we use the simpler DeiT training procedure [56].”
- All in One: Exploring Unified Video-Language Pre-Training (2022, methodology): “We initialize the weights of both self-attention and MLP layers from pre-trained ViT [9] or DeiT [34].”
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, methodology): “The image transformer is initialized from ViT pre-trained on ImageNet (Touvron et al., 2020; Dosovitskiy et al., 2021), and the text transformer is initialized from BERTbase (Devlin et al.”
- Multiview Transformers for Video Recognition (2022, methodology): “We follow [3] and apply the same data augmentation and regularization schemes [13,29,64,82], which were used by [67] to train vision transformers more effectively.”