RegionViT: Regional-to-Local Attention for Vision Transformers

By Chun-Fu Chen, Rameswar Panda, Quanfu Fan. 2021

255 citations recorded by the source corpus.

Vision transformer (ViT) has recently shown its strong capability in achieving comparable results to convolutional neural networks (CNNs) on image classification. However, vanilla ViT simply inherits the same architecture from the natural language processing directly, which is often not optimized for vision applications. Motivated by this, in this paper, we propose a new architecture that adopts the pyramid structure and employ a novel regional-to-local attention rather than global self-attention in vision transformers. More specifically, our model first generates regional tokens and local tokens from an image with different patch sizes, where each regional token is associated with a set of local tokens based on the spatial location. The regional-to-local attention includes two steps: first, the regional self-attention extract global information among all regional tokens and then the local self-attention exchanges the information among one regional token and the associated local tokens via self-attention. Therefore, even though local self-attention confines the scope in a local region but it can still receive global information. Extensive experiments on four vision tasks, including image classification, object and keypoint detection, semantics segmentation and action recognition, show that our approach outperforms or is on par with state-of-the-art ViT variants including many concurrent works. Our source codes and models are available at https://github.com/ibm/regionvit.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Transformers in Vision: A Survey (2021, methodology): “Examples of multi-stage ViTs include Pyramid ViT [208, 209], Twins [42], Co-scale conv-attentional image Transformers (CoaT) [226], Swin Transformer [137], Convolutional vision Transformer (CvT) [218], Shuffle Transformer [87], CrossFormer [210], RegionViT [29], and Focal Transformer models [228].”
CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention (2021, methodology): “The competitors are all competitive vision transformers, including DeiT (Touvron et al., 2021), PVT (Wang et al., 2021), T2T-ViT (Yuan et al., 2021), TNT (Han et al., 2021), CViT (Chen et al., 2021a), Twins (Chu et al., 2021), Swin (Liu et al., 2021b), NesT (Zhang et al., 2021b), CvT (Wu et al., 202”
Vision Transformer with Deformable Attention (2022, methodology): “In order to avoid excessive attention computation, existing works [6, 11, 26, 36, 43, 49] have leveraged carefully designed efficient attention patterns to reduce the computation complexity.”
A Survey on Vision Transformer (2020, background): “Recently, the researchers begin to pay attention to improve the modeling capacity for local information [85], [148], [26].”
CrossFormer: A Versatile Vision Transformer Based on Cross-scale Attention (2021, unknown): “RPB is shown to be more effective than other position representations for visual tasks in previous works (Liu et al., 2021b; Chen et al., 2021b).”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “Researchers thus have made plenty of efforts to decrease computation complexity [153], [156], [161], [164], [168], [176] and memory cost [154], [161], [164], [165], [176].”
EfficientFormer: Vision Transformers at MobileNet Speed (2022, methodology): “…methodologies, such as proposing new architectures or modules [53, 54, 55, 56, 57, 58], re-thinking self-attention and sparse-attention mechanisms [59, 60, 61, 62, 63, 64, 65], and utilizing search algorithms that are widely explored in CNNs to ﬁnd smaller and faster ViTs [66, 28, 29, 67].”
Rethinking Vision Transformers for MobileNet Size and Speed (2022, methodology): “For reducing the computation complexity of ViTs, many works propose new modules and architecture design [40, 29, 23, 45, 65], while others eliminate redundancies in attention mechanism [78, 32, 10, 48, 16, 64, 75, 5].”
Vision Transformers for Single Image Dehazing (2022, methodology): “Subsequently, many modified architectures [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41] have been proposed, and vision Transformer is challenging the dominance of CNNs in high-level vision tasks.”