Incorporating Convolution Designs into Visual Transformers

By Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fisher Yu, Wei Wu. 2021

600 citations recorded by the source corpus.

Motivated by the success of Transformers in natural language processing (NLP) tasks, there emerge some attempts (e.g., ViT and DeiT) to apply Transformers to the vision domain. However, pure Transformer architectures often require a large amount of training data or extra supervision to obtain comparable performance with convolutional neural networks (CNNs). To overcome these limitations, we analyze the potential drawbacks when directly borrowing Transformer architectures from NLP. Then we propose a new Convolution-enhanced image Transformer (CeiT) which combines the advantages of CNNs in extracting low-level features, strengthening locality, and the advantages of Transformers in establishing long-range dependencies. Three modifications are made to the original Transformer: 1) instead of the straightforward tokenization from raw input images, we design an Image-to-Tokens (I2T) module that extracts patches from generated low-level features; 2) the feed-froward network in each encoder block is replaced with a Locally-enhanced Feed-Forward (LeFF) layer that promotes the correlation among neighboring tokens in the spatial dimension; 3) a Layer-wise Class token Attention (LCA) is attached at the top of the Transformer that utilizes the multi-level representations.Experimental results on ImageNet and seven down-stream tasks show the effectiveness and generalization ability of CeiT compared with previous Transformers and state- of-the-art CNNs, without requiring a large amount of training data and extra CNN teachers. Besides, CeiT models demonstrate better convergence with 3× fewer training iterations, which can reduce the training cost significantly 1.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Early Convolutions Help Transformers See Better (2021, background): “Resonating with our work, [5, 17, 47, 49] present models with convolutional stems, but do not analyze optimizability (our focus).”
Towards Robust Vision Transformer (2021, methodology): “Base on this seminal work, numerous variants have been proposed to improve ViTs from different perspectives containing training data efficiency [40], self-attention mechanism [25], introducing convolution [23,45,50] or pooling layers [20,43], etc.”
Transformers in Vision: A Survey (2021, methodology): “Convolution enhanced image Transformers [234] employ convolutions based image-to-token module to extract low-level features.”
CoAtNet: Marrying Convolution and Attention for All Data Sizes (2021, methodology): “In comparison, after the success of ViT and ResNet-ViT [13], another popular line of research starts with a Transformer backbone and tries to incorporate explicit convolution or some desirable properties of convolution into the Transformer backbone [25, 24, 23, 22, 21, 43, 44].”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, background): “…from different perspectives, such as data-efﬁcient training [57], improved patch embedding/encoding [18, 75, 32], integrating convolutional projections into transformers [67, 74], multi-scale architectures and efﬁcient self-attention mechanisms for high-resolution vision tasks [63, 67, 44, 80, 17].”
XCiT: Cross-Covariance Image Transformers (2021, background): “Yuan et al. [78] improve both the performance and the convergence speed of vision transformers by replacing the linear patch projection with convolutional layers and max-pooling, as well as modifying the feed-forward networks in each transformer layer to incorporate depth-wise convolutions.”
ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond (2022, background): “Recently, some works try to introduce the intrinsic IB of CNNs into vision transformers explicitly [26,58,25,43,18,86, 80,91,9,49].”
Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition (2021, background): “Another line of research proposes to integrate the inductive bias of CNN into Transformers [49, 8, 54, 16].”
Review of Image Classification Algorithms Based on Convolutional Neural Networks (2021, background): “There are also plenty of works trying to augment a pure transformer block or self-attention layer with a convolution operation, e.g., BoTNet [224], CeiT [225], CoAtNet [226], CvT [227].”
A Survey of Visual Transformers (2021, methodology): “CeiT [55] and LocalViT [56] extract the locality by directly adding a depth-wise convolution in FFN.”
Fast Vision Transformers with HiLo Attention (2022, background): “Since then, subsequent works have been proposed to improve ViT by incorporating more convolutional layers [43, 49], introducing pyramid feature maps [40, 25], enhancing the locality [50], as well as automatically searching a well-performed architecture [4, 2] with neural architecture search (NAS).”
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer (2021, background): “To achieve better accuracy-speed trade-offs for general dense prediction, recent works (Yuan et al. 2021; Graham et al. 2021; Wang et al. 2021a) design transformers of deep-narrow structures by adopting sub-sampling operation (e.g., strided down-sampling, local average pooling, convolutional…”
DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition (2023, background): “Moreover, some works [10], [27], [61], [69], [82], [85], [96] combine CNNs and Transformers”
P2T: Pyramid Pooling Transformer for Scene Understanding (2021, neutral): “techniques such as patch embedding [51], positional encoding [50], and feed-forward network [24], [52], [53].”
CAT: Cross Attention in Vision Transformer (2021, methodology): “Both CvT[35] and CeiT[51] introduce the convolutional layer to replace the linear projection of QKV[29].”
Residual Mixture of Experts (2022, background): “Triggered by ViT, a series of follow-up works [8, 9, 14, 17, 21, 25, 30, 38, 49, 50, 52, 54, 57, 59, 60] continuously refresh the image classification records and push the potential of vision transformers in computer vision to a new height.”
PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining (2022, methodology): “Following [23], we incorporate a depth-wise convolution into the Feed-Forward module of the ViT architecture, termed Locally-enhanced Feed-Forward module (LeFF), improving the patch-level local perception and interaction.”