BiFormer: Vision Transformer with Bi-Level Routing Attention

By Lei Zhu, Xinjiang Wang, Zhanghan Ke, Wayne Zhang, Rynson W. H. Lau. Computer Vision and Pattern Recognition, 2023

1,025 citations recorded by the source corpus.

As the core building block of vision transformers, attention is a powerful tool to capture long-range dependency. However, such power comes at a cost: it incurs a huge computation burden and heavy memory footprint as pairwise token interaction across all spatial locations is computed. A series of works attempt to alleviate this problem by introducing handcrafted and content-agnostic sparsity into attention, such as restricting the attention operation to be inside local windows, axial stripes, or dilated windows. In contrast to these approaches, we propose a novel dynamic sparse attention via bi-level routing to enable a more flexible allocation of computations with content awareness. Specifically, for a query, irrelevant key-value pairs are first filtered out at a coarse region level, and then fine-grained token-to-token attention is applied in the union of remaining candidate regions (i.e., routed regions). We provide a simple yet effective implementation of the proposed bilevel routing attention, which utilizes the sparsity to save both computation and memory while involving only GPU-friendly dense matrix multiplications. Built with the proposed bi-level routing attention, a new general vision transformer, named BiFormer, is then presented. As BiFormer attends to a small subset of relevant tokens in a query adaptive manner without distraction from other irrelevant ones, it enjoys both good performance and high computational efficiency, especially in dense prediction tasks. Empirical results across several computer vision tasks such as image classification, object detection, and semantic segmentation verify the effectiveness of our design. Code is available at https://github.com/rayleizhu/BiFormer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2f4d8f3c016ec53380b376ae7ac516f9c0f07a0d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zijia Lin
2. Hui Chen
3. Ao Wang
4. Hanyu Hong
5. Pei An
6. G. Wang
7. Yanfei Chen
8. Tiange Huang

In Computer Vision and Pattern Recognition

1. Zijia Lin
2. Hui Chen
3. Ao Wang
4. Guiguang Ding
5. Hanyu Hong
6. Pei An
7. G. Wang
8. Yanfei Chen

Citations in the corpus

• UAV-YOLOv8: A Small-Object-Detection Model Based on Improved YOLOv8 for UAV Aerial Photography Scenarios (2023, methodology): “[30] proposed a novel dynamic sparse attention: the Bi-Level Routing Attention, whose workflow is shown in Figure 4a.”
• Rep ViT: Revisiting Mobile CNN From ViT Perspective (2023, background): “Building on the competitive performance of ViTs, subsequent works have sought to incorporate spatial inductive biases to enhance their stability and performance [10, 21], design more efficient self-attention operations [16, 75], and adapt ViTs to a diverse range of computer vision tasks [18, 70].”
• TransNeXt: Robust Foveal Visual Perception for Vision Transformers (2023, neutral): “Moreover, under similar throughput conditions, our model demonstrates competitive top-1 accuracy on ImageNet-1K compared to previous state-of-the-art models such as MaxViT [25], BiFormer [34], and QuadTree [24].”