CvT: Introducing Convolutions to Vision Transformers

By Haiping Wu, Bin Xiao, Noel C. F. Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, Lei Zhang. 2021

2,478 citations recorded by the source corpus.

We present in this paper a new architecture, named Convolutional vision Transformer (CvT), that improves Vision Transformer (ViT) in performance and efficiency by introducing convolutions into ViT to yield the best of both de-signs. This is accomplished through two primary modifications: a hierarchy of Transformers containing a new convolutional token embedding, and a convolutional Transformer block leveraging a convolutional projection. These changes introduce desirable properties of convolutional neural networks (CNNs) to the ViT architecture (i.e. shift, scale, and distortion invariance) while maintaining the merits of Transformers (i.e. dynamic attention, global context, and better generalization). We validate CvT by conducting extensive experiments, showing that this approach achieves state-of-the-art performance over other Vision Transformers and ResNets on ImageNet-1k, with fewer parameters and lower FLOPs. In addition, performance gains are maintained when pretrained on larger datasets (e.g. ImageNet-22k) and fine-tuned to downstream tasks. Pretrained on ImageNet-22k, our CvT-W24 obtains a top-1 accuracy of 87.7% on the ImageNet-1k val set. Finally, our results show that the positional encoding, a crucial component in existing Vision Transformers, can be safely re-moved in our model, simplifying the design for higher resolution vision tasks. Code will be released at https://github.com/microsoft/CvT.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

Early Convolutions Help Transformers See Better (2021, background): “Resonating with our work, [5, 17, 47, 49] present models with convolutional stems, but do not analyze optimizability (our focus).”
VOLO: Vision Outlooker for Visual Recognition (2021, background): “Some of them [4, 19, 61, 54, 79] introduce local dependency into vision transformers by modifying the patch embedding block or the transformer block or both, while others [21, 37, 57] adopt a pyramid structure to reduce the overall computation while maintaining the models’ ability to capture low-lev”
Towards Robust Vision Transformer (2021, background): “from different perspectives containing training data efficiency [40], self-attention mechanism [25], introducing convolution [23,45,50] or pooling layers [20,43], etc.”
Florence: A New Foundation Model for Computer Vision (2021, methodology): “Specifically, we replace the patch embedding and patch merging modules in the Swin Transformer (Liu et al., 2021a) with the convolutional embedding layers as described in CvT (Wu et al., 2021).”
Primer: Searching for Efficient Transformers for Language Modeling (2021, methodology): “MDHA is similar to Convolutional Attention [43], which uses separable convolution instead of depthwise convolution and does not apply convolution operations per attention head as we do.”
Unified Contrastive Learning in Image-Text-Label Space (2022, methodology): “With this goal, numerous works have pushed the image recognition performance from different directions, such as data scale from MNIST [36] to ImageNet1K [10], model architectures from convolutional neural networks (CNNs) [23, 26, 34, 35, 40, 52, 53] to Transformers [15, 43, 57, 61, 64, 67, 72], and ”
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers (2021, background): “In [22, 13, 36, 38] the authors propose hierarchical versions of ViT.”
Online Continual Learning with Contrastive Vision Transformer (2022, methodology): “Considering the superior modeling capability of Vision Transformers [21] recently demonstrated on computer vision tasks [57,84,26,92,17], we take the utilization of the potential of attention mechanism [74] to develop online CL.”
Transformers in Vision: A Survey (2021, methodology): “For example, Convolutional vision Transformer (CvT) [219] incorporate convolution based projection to capture the spatial structure and low-level details, for tokenization of image patches.”
CoAtNet: Marrying Convolution and Attention for All Data Sizes (2021, methodology): “In comparison, after the success of ViT and ResNet-ViT [13], another popular line of research starts with a Transformer backbone and tries to incorporate explicit convolution or some desirable properties of convolution into the Transformer backbone [25, 24, 23, 22, 21, 43, 44].”
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (2021, methodology): “After that, methods such as Swin [9], CvT [58], CoaT [59], LeViT [60] and Twins [10] enhance the local continuity of features and remove ﬁxed size position embedding to improve the performance of Transformers in dense prediction tasks.”
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference (2021, background): “Two architecture that comes close to LeViT are the poolingbased vision transformer (PiT) [59] and CvT [60], ViT variants with a pyramid structure.”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, methodology): “On one hand, a number of works in both NLP and CV resort to coarse-grained global self-attention by attending the downsampled/summarized tokens, while preserving the long-range interactions [50, 47, 63, 67, 32].”
PVT v2: Improved baselines with Pyramid Vision Transformer (2021, methodology): “Recent studies on vision Transformer are converging on the backbone network [7, 29, 31, 32, 21, 34, 9, 4] designed for downstream vision tasks, such as image classification, object detection, instance and semantic segmentation.”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, methodology): “Thus, instead of organizing visual tokens into grids, as recent ViT-based applications [16, 23, 44, 81] do, we propose to perform hierarchical grouping of visual tokens into irregular-shaped segments.”
HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions (2022, background): “Many enhanced vision Transformers also use lightweight convolutions in the basic building block to efficiently capture neighboring patterns [15, 56, 17] or relax the quadratic complexity of self-attention [9, 54, 18].”
AutoFormer: Searching Transformers for Visual Recognition (2021, background): “There are two concurrent works, i.e. , Boss-NAS [32] and CvT [60], exploring different search space from ours.”
CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows (2021, methodology): “Vision Transformers.”