Vision Transformer with Deformable Attention

By Zhuofan Xia, Xuran Pan, Shiji Song, L. Li, Gao Huang. Computer Vision and Pattern Recognition, 2022

822 citations recorded by the source corpus.

Transformers have recently shown superior performances on various vision tasks. The large, sometimes even global, receptive field endows Transformer models with higher representation power over their CNN counterparts. Nevertheless, simply enlarging receptive field also gives rise to several concerns. On the one hand, using dense attention e.g., in ViT, leads to excessive memory and computational cost, and features can be influenced by irrelevant parts which are beyond the region of interests. On the other hand, the sparse attention adopted in PVT or Swin Transformer is data agnostic and may limit the ability to model long range relations. To mitigate these issues, we propose a novel deformable selfattention module, where the positions of key and value pairs in selfattention are selected in a data-dependent way. This flexible scheme enables the self-attention module to focus on relevant re-gions and capture more informative features. On this basis, we present Deformable Attention Transformer, a general backbone model with deformable attention for both image classification and dense prediction tasks. Extensive experi-ments show that our models achieve consistently improved results on comprehensive benchmarks. Code is available at https://github.com/LeapLabTHU/DAT.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/e5cb26148791b57bfd36aa26ce2401e231d01b57

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Xiaogang Wang
3. Dacheng Tao
4. Yu Qiao
5. Ming-Hsuan Yang
6. Hongsheng Li
7. Alan L. Yuille
8. Ping Luo

In Electrical and Electronic Engineering

1. Yu Qiao
2. Shiji Song
3. Gao Huang
4. Jianfei Cai
5. Rynson W. H. Lau
6. Xiaogang Wang
7. Ping Luo
8. Hongsheng Li

Citations in the corpus

• FLatten Transformer: Vision Transformer using Focused Linear Attention (2023, methodology): “With the advent of Vision Transformer [11, 39], self-attention techniques have shown great potential in a variety of vision tasks including image classification [41, 43, 30, 46], semantic segmentation [6, 49], object detection [4, 61, 22], and multi-modal tasks [35, 31].”
• ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond (2022, neutral): “, 2021), DAT-T (Xia et al., 2022), and our ViTAEv2-S.”
• HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation (2022, background): “Several derivatives of vision transformers have been introduced to make them more efficient and less dependent on a large-sized dataset to achieve generalization [44, 55, 51].”
• InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions (2022, methodology): “To address this problem, PVT [10, 11] and Linformer [45] performed global attention on the downsampled key and value maps, DAT [46] employed deformable attention to sparsely sample information from value maps, while HaloNet [47] and Swin transformer [2] developed local attention mechanisms and used ”
• Rethinking Vision Transformers for MobileNet Size and Speed (2022, background): “The works that do not have public models or are not compatible with mobile devices [6, 82, 51] are not contained in Tab.”
• BiFormer: Vision Transformer with Bi-Level Routing Attention (2023, background): “(e) Deformable attention [48] enables image-adaptive sparsity via deforming a regular grid.”
• PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation (2024, background): “( 2) Local Attention [14, 24, 48, 56] focuses attention within nearby regions; notably, Swin Transformer [24] utilizes window-based attention to limit computations to within a specified window size.”
• Fast Vision Transformers with HiLo Attention (2022, background): “In Figure 6, we show that HiLo is consistently faster than many attention mechanisms [51, 32, 1, 50, 11, 17, 55, 60, 41, 44, 20, 18] on both CPUs and GPUs.”