CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification

By Chun-Fu Chen, Quanfu Fan, Rameswar Panda. 2021

2,175 citations recorded by the source corpus.

The recently developed vision transformer (ViT) has achieved promising results on image classification compared to convolutional neural networks. Inspired by this, in this paper, we study how to learn multi-scale feature representations in transformer models for image classification. To this end, we propose a dual-branch transformer to com-bine image patches (i.e., tokens in a transformer) of different sizes to produce stronger image features. Our approach processes small-patch and large-patch tokens with two separate branches of different computational complexity and these tokens are then fused purely by attention multiple times to complement each other. Furthermore, to reduce computation, we develop a simple yet effective token fusion module based on cross attention, which uses a single token for each branch as a query to exchange information with other branches. Our proposed cross-attention only requires linear time for both computational and memory complexity instead of quadratic time otherwise. Extensive experiments demonstrate that our approach performs better than or on par with several concurrent works on vision transformer, in addition to efficient CNN models. For example, on the ImageNet1K dataset, with some architectural changes, our approach outperforms the recent DeiT by a large margin of 2% with a small to moderate increase in FLOPs and model parameters. Our source codes and models are available at https://github.com/IBM/CrossViT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Multiview Transformers for Video Recognition (2022, background): “In practice, due to the limited amount of training data, applying similar multiscale processing in transformers [11, 21, 44, 74] to reduce complexity of the model has proven to be effective.”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “For image recognition, vision Transform-ers [9, 17, 45, 66, 68, 71, 73, 76] replace CNNs by an image patch tokenizer and a transformer encoder, which have been proved to obtain competitive performance as CNNs. [4, 5, 74] make attempts to apply Transformers on video recognition in a convolution-free…”
Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “Recently, transformer architectures [7, 26, 86, 97, 104, 109] shine in semantic segmentation, focusing on multi-scale feature fusion [12, 84, 89, 101] and contextual feature aggregation [58, 79].”
Transformers in Vision: A Survey (2021, neutral): “CrossViT [104] encodes multi-scale features with two branches (each with multiple transformer blocks), by separately processesing smaller and larger image patches.”
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (2021, methodology): “More recent methods such as T2T ViT [53], CPVT [54], TNT [55], CrossViT [56] and LocalViT [57] introduce tailored changes to ViT to further improve image classiﬁcation performance.”
MPViT: Multi-Path Vision Transformer for Dense Prediction (2021, methodology): “9 Small (S) [1, 3, 6, 3] [ 64, 128, 216, 288] 22.”
MambaVision: A Hybrid Mamba-Transformer Vision Backbone (2024, neutral): “CoAT [36] and CrossViT [1] demonstrated enhanced feature learning by combining convolutions with self-attention, while NextViT [ 17] systematically incorporated CNN-like processing into Transformers.”
DS-TransUNet: Dual Swin Transformer U-Net for Medical Image Segmentation (2021, background): “The close works include: CrossViT [21] proposes a dualbranch transformer and cross-attention for image classification.”
Vision Transformers for Single Image Dehazing (2022, methodology): “Subsequently, many modified architectures [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41] have been proposed, and vision Transformer is challenging the dominance of CNNs in high-level vision tasks.”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “The subﬁelds include point cloud segmentation, domain-aware segmentation, label and model efﬁcient segmentation, class agnostic segmentation, tracking, and medical segmentation.”
HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation (2022, methodology): “HiFormer-S ResNet34 96 [2,2,6] [3,6,12] 7 384 96 1 1 1 3 HiFormer-B ResNet50 96 [2,2,6] [3,6,12] 7 384 96 2 1 2 6 HiFormer-L ResNet34 96 [2,2,6] [3,6,12] 7 384 96 4 1 4 6”
EfficientFormer: Vision Transformers at MobileNet Speed (2022, methodology): “To accelerate ViT, many approaches have been introduced with different methodologies, such as proposing new architectures or modules [53, 54, 55, 56, 57, 58], re-thinking self-attention and sparse-attention mechanisms [59, 60, 61, 62, 63, 64, 65], and utilizing search algorithms that are widely…”
Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting (2023, methodology): “Transformer is initially applied to NLP for sequence modeling, recent works apply transformer to CV tasks to process images (Dosovitskiy et al., 2021; Touvron et al., 2021; Liu et al., 2021b; Chen et al., 2021; Han et al., 2021).”