Focal Self-attention for Local-Global Interactions in Vision Transformers

By Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, Jianfeng Gao. 2021

538 citations recorded by the source corpus.

Recently, Vision Transformer and its variants have shown great promise on various computer vision tasks. The ability of capturing short- and long-range visual dependencies through self-attention is arguably the main source for the success. But it also brings challenges due to quadratic computational overhead, especially for the high-resolution vision tasks (e.g., object detection). In this paper, we present focal self-attention, a new mechanism that incorporates both fine-grained local and coarse-grained global interactions. Using this new mechanism, each token attends the closest surrounding tokens at fine granularity but the tokens far away at coarse granularity, and thus can capture both short- and long-range visual dependencies efficiently and effectively. With focal self-attention, we propose a new variant of Vision Transformer models, called Focal Transformer, which achieves superior performance over the state-of-the-art vision Transformers on a range of public image classification and object detection benchmarks. In particular, our Focal Transformer models with a moderate size of 51.1M and a larger size of 89.8M achieve 83.5 and 83.8 Top-1 accuracy, respectively, on ImageNet classification at 224x224 resolution. Using Focal Transformers as the backbones, we obtain consistent and substantial improvements over the current state-of-the-art Swin Transformers for 6 different object detection methods trained with standard 1x and 3x schedules. Our largest Focal Transformer yields 58.7/58.9 box mAPs and 50.9/51.3 mask mAPs on COCO mini-val/test-dev, and 55.4 mIoU on ADE20K for semantic segmentation, creating new SoTA on three of the most challenging computer vision tasks.

Subfields: Aerospace Engineering, Cognitive Neuroscience, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Florence: A New Foundation Model for Computer Vision (2021, methodology): “For the image encoder, we chose hierarchical Vision Transformers (e.g. , Swin (Liu et al., 2021a), CvT (Wu et al., 2021), Vision Longformer (Zhang et al., 2021a), Focal Transformer (Yang et al., 2021), and CSwin (Dong et al., 2021)).”
Grounded Language-Image Pre-training (2021, methodology): “We ﬁrst choose the SoTA detection model Dynamic Head (Dy-Head) [10] based on the Swin-Tiny Transformer backbone [39] as the base object detection mode.”
CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers (2022, background): “Recent works have been focused on improving the architectures of attention models, including sparse attention [33, 34, 35, 36, 37, 38, 39], enlarged receptive fields [40, 41], pyramidal designs [42, 43, 44], efficient alternatives [45, 46, 47], etc.”
Uformer: A General U-Shaped Transformer for Image Restoration (2021, background): “Rather than focusing on image classiﬁcation, recent works [10,13,22,67,82] propose a brunch of Transformer-based backbones for more general high-level vision tasks.”
MaxViT: Multi-Axis Vision Transformer (2022, background): “To properly regularize the model capacity and improve its scalability, numerous subsequent efforts have studied sparse Transformer models tailored for vision tasks such as local attention [12, 48, 54, 93].”
UniFormer: Unifying Convolution and Self-Attention for Visual Recognition (2022, methodology): “For example, our UniFormer-B outperforms the powerful CSwin-S [22] by +0.3 box mAP and +0.3 mask mAP, and even better than larger backbones such as Swin-B and Focal-B [108].”
Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks (2023, background): “Thanks to the strong universal representation learned by Florence-2 pre-training, we do not require longer training epochs, such as 36 epochs in [51, 81, 85, 86], or 100 epochs in [46], to achieve better results.”
Fast Vision Transformers with HiLo Attention (2022, methodology): “Second, some efficient attention mechanisms in ViTs have low theoretical complexity guarantee but are actually slow on GPUs due to particular operations that are not hardware-friendly or cannot be parallelized, such as the multi-scale window partition [48], recursion [35] and dilated window [15].”
Earthformer: Exploring Space-Time Transformers for Earth System Forecasting (2022, background): “Focal transformer [49] proposes focal self-attention that can incorporate both fine-grained local and coarse-grained global interactions.”
Stratified Transformer for 3D Point Cloud Segmentation (2022, methodology): “Methods of [9, 12, 57] [34,35] use max pooling to aggregate features.”
InceptionNeXt: When Inception Meets ConvNeXt (2023, unknown): “Many ViT variants [15, 36, 39, 64, 67, 75, 79], like DeiT [64] and Swin [39], are proposed and have achieved remarkable performance across a wide range of vision tasks.”
Vision Transformer Adapter for Dense Predictions (2022, methodology): “From Table 3, we find that our ViT-Adapter achieves comparable accuracy with specific-designed transformers [35,52,75,79].”
SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation (2022, background): “Speaking of the recent popular vision transformers [17, 51, 82, 74, 73, 50, 80, 33, 49, 88], they usually ignore adaptability in channel dimension.”
VMamba: Visual State Space Model (2024, methodology): “With the aid of powerful computing devices ( GPU ) and large-scale datasets [8], increasingly deeper [42, 46, 22, 25] and efficient models [23, 47, 59, 41] have been proposed to enhance performance across a spectrum of visual tasks.”