CMT: Convolutional Neural Networks Meet Vision Transformers

By Jianyuan Guo, Kai Han, Han Wu, Chang Xu, Yehui Tang, Chunjing Xu, Yunhe Wang. Computer Vision and Pattern Recognition, 2021

951 citations recorded by the source corpus.

Vision transformers have been successfully applied to image recognition tasks due to their ability to capture long-range dependencies within an image. However, there are still gaps in both performance and computational cost between transformers and existing convolutional neural networks (CNNs). In this paper, we aim to address this issue and develop a network that can outperform not only the canonical transformers, but also the high-performance convolutional models. We propose a new transformer based hybrid network by taking advantage of transformers to capture long-range dependencies, and of CNNs to extract local information. Furthermore, we scale it to obtain a family of models, called CMTs, obtaining much better trade-off for accuracy and efficiency than previous CNN-based and transformer-based models. In particular, our CMT-S achieves 83.5% top-1 accuracy on ImageNet, while being 14x and 2x smaller on FLOPs than the existing DeiT and EfficientNet, respectively. The proposed CMT-S also generalizes well on CIFAR10 (99.2%), CIFAR100 (91.7%), Flowers (98.7%), and other challenging vision datasets such as COCO (44.3% mAP), with considerably less computational cost.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/761240b06248b9836ee564bdab61559c84b681ed

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

HiFuse: Hierarchical Multi-Scale Feature Fusion Network for Medical Image Classification (2022, methodology): “The models, such as VitAE, StoHis, TransFuse, CMT, Conformer and so on,Xu et al. [2021], Fu et al. [2022], Zhang et al. [2021], Guo et al. [2022a], Peng et al. [2021] not only inherit the structural advantages of CNN and Transformer, but also verify that the coupling of local features and global…”
DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition (2023, background): “Moreover, some works [40], [41], [74], [75], [76], [77], [78] combine CNNs and Transformers for introducing the locality prior, and they usually design hand-crafted and complex modules for interaction between CNNs and Transformers features, leading to a lack of scalability to large-scale parameters…”
MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, methodology): “CMT [14] block contains depthwise convolution layers based local perception unit and a lightweight transformer block.”
MetaFormer is Actually What You Need for Vision (2021, background): “A lot of works have been focusing on improving the token mixing approach of transformers by shifted windows [35], relative position encoding [59], refining attention map [66], or incorporating convolution [12, 20, 58], etc.”
Rep ViT: Revisiting Mobile CNN From ViT Perspective (2023, background): “Building on the competitive performance of ViTs, subsequent works have sought to incorporate spatial inductive biases to enhance their stability and performance [8,17], design more efficient self-attention operations by regularizing the pattern of attention mechanisms [13, 69], and adapt ViTs to a d”
EfficientFormer: Vision Transformers at MobileNet Speed (2022, methodology): “Inspired by the competitive performance and global receptive field of transformer models, follow-up works are proposed to refine the architecture [36, 37], explore the relationship between CONV nets and ViT [38, 39, 40], and adapt ViT to different computer vision tasks [13, 41, 42, 43, 44, 45, 46].”
LocalViT: Analyzing Locality in Vision Transformers (2021, positive): “In the meanwhile, we also noticed some other works introducing convolutions to different parts of transformers [17], [12].”
Vision Transformer with Deformable Attention (2022, methodology): “Without inserting convolutions in Transformer blocks [13, 14, 35], or using overlapped convolutions in patch embeddings [6, 11, 45], DATs achieve gains of +0.7, +0.7 and +0.5 over Swin Transformer [26] counterparts.”
Focal Modulation Networks (2022, methodology): “Afterwards, multi-scale architectures [5, 69, 78], light-weight convolution layers [74, 22, 39], local self-attention mechanisms [46, 87, 14, 79] and learnable attention weights [84] have been proposed to boost the performance and support high-resolution input.”
PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining (2022, background): “Besides, as pointed out in [22, 23], the standard ViT may not fully leverage the local context information, which limits the visual representation capacity of the ViT-based image encoder.”
Rethinking Vision Transformers for MobileNet Size and Speed (2022, methodology): “Later research further improves ViTs by incorporating hierarchical design [77, 74], injecting locality with the aid of convolutions [26, 18, 28, 68], or exploring different types of token mixing such as local attention [52, 20], spatial MLP mixer [71, 70], and non-parameterized pool mixer [86].”
Recipe for a General, Powerful, Scalable Graph Transformer (2022, methodology): “Considering the great successes of Transformers in natural language processing (NLP) [52, 30] and recently also in computer vision [16, 23, 22], it is natural to study their applicability in the graph domain as well.”
Gold-YOLO: Efficient Object Detector via Gather-and-Distribute Mechanism (2023, background): “As described in the survey [20], [26, 27, 59, 21] has achieved a balance between precision and speed, while [9, 35, 22, 18] has shown strong performance in precision.”
How Do Vision Transformers Work? (2022, methodology): “In addition, prior works observed that MSAs have the following intriguing properties: 1 MSAs improve the predictive performance of CNNs (Dai et al., 2021; Guo et al., 2021; Srinivas et al., 2021), ar X iv :2 20 2.”
Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimation (2022, methodology): “With the rise of vision transformer (ViT) [8] recent work applied it to various computer vision tasks [4, 16, 29, 31, 34], and achieved promising results.”
Transformer-Based Visual Segmentation: A Survey (2023, background): “Meanwhile, several works [139], [140], [141], [142] directly add CNN layers to a transformer to explore the local context.”
FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization (2023, methodology): “Recent hybrid architectures [42, 17, 10, 64] effectively”
FLatten Transformer: Vision Transformer using Focused Linear Attention (2023, methodology): “CMT [12] combines Transformer blocks with efficient convolution operators like depthwise convolution [37], and achieves better efficiencyperformance trade-off.”