DaViT: Dual Attention Vision Transformers

By Mingyu Ding, Bin Xiao, Noel C. F. Codella, Ping Luo, Jingdong Wang, Lu Yuan. 2022

406 citations recorded by the source corpus.

In this work, we introduce Dual Attention Vision Transformers (DaViT), a simple yet effective vision transformer architecture that is able to capture global context while maintaining computational efficiency. We propose approaching the problem from an orthogonal angle: exploiting self-attention mechanisms with both"spatial tokens"and"channel tokens". With spatial tokens, the spatial dimension defines the token scope, and the channel dimension defines the token feature dimension. With channel tokens, we have the inverse: the channel dimension defines the token scope, and the spatial dimension defines the token feature dimension. We further group tokens along the sequence direction for both spatial and channel tokens to maintain the linear complexity of the entire model. We show that these two self-attentions complement each other: (i) since each channel token contains an abstract representation of the entire image, the channel attention naturally captures global interactions and representations by taking all spatial positions into account when computing attention scores between channels; (ii) the spatial attention refines the local representations by performing fine-grained interactions across spatial locations, which in turn helps the global information modeling in channel attention. Extensive experiments show our DaViT achieves state-of-the-art performance on four different tasks with efficient computations. Without extra data, DaViT-Tiny, DaViT-Small, and DaViT-Base achieve 82.8%, 84.2%, and 84.6% top-1 accuracy on ImageNet-1K with 28.3M, 49.7M, and 87.9M parameters, respectively. When we further scale up DaViT with 1.5B weakly supervised image and text pairs, DaViT-Gaint reaches 90.4% top-1 accuracy on ImageNet-1K. Code is available at https://github.com/dingmyu/davit.

Subfields: Computer Vision and Pattern Recognition, Neurology, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Neurology

In Model Architectures

Citations in the corpus

GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “[21] Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan.”
Focal Modulation Networks (2022, background): “Meanwhile, a number of models have been proposed by augmenting SA with (depth-wise) convolutions to capture long-range dependencies with a good awareness of local structures [74, 22, 78, 20, 18, 35, 7, 17].”
VMamba: Visual State Space Model (2024, methodology): “Specifically, in V6 , we introduced MLP into VMamba due to its computational efficiency, discarded the depth-wise convolution operation, and changed the layer configuration of VMamba-Tiny from [2,2,9,2] to [2,2,2,2] .”
Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks (2023, methodology): “We employ DaViT [20] as the vision encoder.”
YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information (2024, methodology): “…of parameters utilization. in the field of deep learning have mainly focused on how to develop more powerful system architectures and learning methods, such as CNNs [21–23, 42, 55, 71, 72], Trans-formers [8, 9, 40, 41, 60, 69, 70], Perceivers [26, 26, 32, 52, 56, 81, 81], and Mambas [17, 38, 80].”
Segment Everything Everywhere All at Once (2023, methodology): “For the vision backbone, we use FocalT [54] and DaViT-d3 (B) [9].”
Dual Aggregation Transformer for Image Super-Resolution (2023, background): “DaViT [11] proposes dual self-attention to capture global context with linear complexity.”
InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions (2022, background): “Some researchers developed large-scale hybrid ViTs [20, 49] by combining the advantages of ViTs and CNNs at different levels.”
Distract Your Attention: Multi-Head Cross Attention Network for Facial Expression Recognition (2021, background): “Very recently, seminal works based on self-attention mechanisms have emerged, such as Swin-Transformer [26], DAB-DETR [27], DaViT [28], and QFormer [29].”