LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference

By Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Herv'e J'egou, Matthijs Douze. 2021

1,047 citations recorded by the source corpus.

We design a family of image classification architectures that optimize the trade-off between accuracy and efficiency in a high-speed regime. Our work exploits recent findings in attention-based architectures, which are competitive on highly parallel processing hardware. We revisit principles from the extensive literature on convolutional neural networks to apply them to transformers, in particular activation maps with decreasing resolutions. We also introduce the attention bias, a new way to integrate positional information in vision transformers.As a result, we propose LeViT: a hybrid neural network for fast inference image classification. We consider different measures of efficiency on different hardware platforms, so as to best reflect a wide range of application scenarios. Our extensive experiments empirically validate our technical choices and show they are suitable to most architectures. Overall, LeViT significantly outperforms existing convnets and vision transformers with respect to the speed/accuracy tradeoff. For example, at 80% ImageNet top-1 accuracy, LeViT is 5 times faster than EfficientNet on CPU. We release the code at https://github.com/facebookresearch/LeViT.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

Online Continual Learning with Contrastive Vision Transformer (2022, methodology): “Since then, a lot of effort has been dedicated to improving Vision Transformers’ data efficiency and model efficiency [92,32,91,40], where an effective direction is to strategically integrate properties of convolution into the Transformer architecture [73,91,48,84,26,94,23,17].”
Transformers in Vision: A Survey (2021, methodology): “LeViT [72] (name inspired from LeNet [111]) applies a four-layered CNN block (with”
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (2021, methodology): “After that, methods such as Swin [9], CvT [58], CoaT [59], LeViT [60] and Twins [10] enhance the local continuity of features and remove ﬁxed size position embedding to improve the performance of Transformers in dense prediction tasks.”
PVT v2: Improved baselines with Pyramid Vision Transformer (2021, methodology): “After that, Swin Transformer [23], CoaT [36], LeViT [10], and Twins [5] further improve the classiﬁcation, detection, and segmentation performance with Transformer backbones.”
XCiT: Cross-Covariance Image Transformers (2021, background): “[26] with convolutional patch projection layers.”
RegionViT: Regional-to-Local Attention for Vision Transformers (2021, background): “…parallel thread for improving vision transformer is in incorporating CNN-like pyramid structure into ViT (Wu et al., 2021; Wang et al., 2021; Heo et al., 2021; Zhang et al., 2021a; Li et al., 2021; Graham et al., 2021; Liu et al., 2021; Chu et al., 2021a; Yan et al., 2021; Chen et al., 2021b).”
Inception Transformer (2022, background): “LV-ViT [50], LeViT [54] and ViTC [21] further stack several layers of convolutions as the stem for models, which is found helpful in training and achieving better performance.”
Token Merging: Your ViT But Faster (2022, methodology): “, 2019), and some attempt to infuse domain-specific modules (Mehta & Rastegari, 2021; Graham et al., 2021; Liu et al., 2021; 2022b; Dong et al., 2022).”
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, methodology): “3.3.”
Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results (2022, methodology): “Cutout is more prevalent when training CNNs and Mobile-oriented models [17, 15], while a combination of Mixup and Cutmix, as proposed in [41], is popular for training Transformer-based models [11, 38].”
Continual Learning with Lifelong Vision Transformer (2022, methodology): “Unlike vanilla self-attention in vision transformers [21, 25, 32, 74] that derives the attention map by computing similarity between self-queries and self-keys, we propose intertask attention mechanism to obtain attention map by computing the affinities between self-queries and a learnable external ”
Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation (2021, methodology): “We do not use attention operations in the stem since early convolutions are more effective to extract low-level features than self-attentions [14,32].”
ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond (2022, background): “For example, (Li et al., 2021; Graham et al., 2021; Wu et al., 2021; Dai et al., 2021) stack convolutions and attention layers sequentially, resulting in a serial structure and modeling the locality and global dependency accordingly.”
Shunted Self-Attention via Multi-Scale Token Aggregation (2021, background): “Patch Embedding Many recent works [9, 10, 24] study the function of the image to token mapping, i.”
Rep ViT: Revisiting Mobile CNN From ViT Perspective (2023, methodology): “For our RepViT block, we set the expansion ratio to 2 in the channel mixer for all stages, following [20, 29, 37].”