Going deeper with Image Transformers

By Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, Herv'e J'egou. 2021

1,349 citations recorded by the source corpus.

Transformers have been recently adapted for large scale image classification, achieving high scores shaking up the long supremacy of convolutional neural networks. However the optimization of vision transformers has been little studied so far. In this work, we build and optimize deeper transformer networks for image classification. In particular, we investigate the interplay of architecture and optimization of such dedicated transformers. We make two architecture changes that significantly improve the accuracy of deep transformers. This leads us to produce models whose performance does not saturate early with more depth, for in-stance we obtain 86.5% top-1 accuracy on Imagenet when training with no external data, we thus attain the current sate of the art with less floating-point operations and parameters. Our best model establishes the new state of the art on Imagenet with Reassessed labels and Imagenet-V2 / match frequency, in the setting with no additional training data. We share our code and models1.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Benchmarking Detection Transfer Learning with Vision Transformers (2021, methodology): “(3) BEiT makes use of layer scale [37] in pre-training, while the other methods do not.”
Early Convolutions Help Transformers See Better (2021, background): “It has been noted in the literature that ViT models are challenging to optimize, e.g ., they may achieve only modest performance when trained on a mid-size dataset (ImageNet-1k) [13], are sensitive to data augmentation [41] and optimizer choice [41], and may perform poorly when made deeper [42].”
VOLO: Vision Outlooker for Visual Recognition (2021, background): “6 CaiT-S36↑384 [52] Transformer 68M 48.”
Stable and low-precision training for large-scale vision-language models (2023, methodology): “All methods we try diverge except for using zero-init layerscale [56], which multiplies the output of each self-attention or mlp block with a learnable vector initialized to zero.”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “Transformers have enabled a uniﬁed architecture paradigm for different modalities, which only need the modality-speciﬁc tokenizers to convert inputs from different modalities into uniﬁed input token sequences.”
Scaling Vision Transformers (2021, background): “MAP head has also been explored in [41], in a different context for better quality rather than saving memory.”
CoAtNet: Marrying Convolution and Attention for All Data Sizes (2021, methodology): “Subsequent works use special regularization and stronger data augmentation to improve the vanilla ViT [16, 17, 18], yet none of these ViT variants could outperform the SOTA convolution-only models on ImageNet classification given the same amount of data and computation [19, 20].”
A ConvNet for the 2020s (2022, methodology): “Layer Scale [74] of initial value 1e-6 is applied.”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, methodology): “Even without multi-scale design as in CNNs, a number of Transformer-based works such as TNT [32], DeepViT [89] and CaiT [58] achieve superior performance to CNN models with comparable model size and computational cost.”
XCiT: Cross-Covariance Image Transformers (2021, methodology): “In particular, our XCiT-L24 model achieves 86.0% top-1 accuracy on ImageNet, outperforming its CaiT-M24 [67] and NFNet-F2 [10] counterparts with comparable numbers of parameters.”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, methodology): “Inspired by the success of Transformers in NLP [20,76], the Vision Transformer (ViT) [22] was recently proposed and has been successfully applied to multiple computer vision tasks, including image classification [44, 72, 73, 86], object detection [44, 79, 88], semantic segmentation [44, 82, 91] and ”
EVA-02: A Visual Representation for Neon Genesis (2023, background): “Although the inner-block micro architecture of plain ViT has continuously evolved since its inception in the year 2020 [109, 117], we notice that some significant architectural advances in language models have not yet been explored in the context of visual representation learning.”
UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning (2022, methodology): “Vision Transformers (Dosovitskiy et al., 2021; Touvron et al., 2021a;b; Liu et al., 2021a) have been popular for vision tasks and outperform many CNNs. Based on ViT, several prior works (Bertasius et al., 2021; Neimark et al., 2021; Sharir et al., 2021; Li et al., 2021b; Arnab et al., 2021; Bulat…”
Unlocking High-Accuracy Differentially Private Image Classification through Scale (2022, methodology): “…with group normalization (Wu and He, 2020), using large batch sizes, weight standardization (Qiao et al., 2019), a modiﬁcation to DP-SGD which we call augmentation multiplicity (Fort et al., 2021; Hoﬀer et al., 2019; Touvron et al., 2021a) and parameter averaging (Polyak and Juditsky, 1992).”