Twins: Revisiting the Design of Spatial Attention in Vision Transformers

By Xiangxiang Chu, Zhi Tian, Yuqing Wang, Haibing Ren, Xiaolin Wei, Huaxia Xia, Chunhua Shen. 2021

1,341 citations recorded by the source corpus.

Very recently, a variety of vision transformer architectures for dense prediction tasks have been proposed and they show that the design of spatial attention is critical to their success in these tasks. In this work, we revisit the design of the spatial attention and demonstrate that a carefully-devised yet simple spatial attention mechanism performs favourably against the state-of-the-art schemes. As a result, we propose two vision transformer architectures, namely, Twins-PCPVT and Twins-SVT. Our proposed architectures are highly-efficient and easy to implement, only involving matrix multiplications that are highly optimized in modern deep learning frameworks. More importantly, the proposed architectures achieve excellent performance on a wide range of visual tasks, including image level classification as well as dense detection and segmentation. The simplicity and strong performance suggest that our proposed architectures may serve as stronger backbones for many vision tasks. Our code is released at https://github.com/Meituan-AutoML/Twins .

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Geology

Citations in the corpus

Transformers in Vision: A Survey (2021, methodology): “Twins [37] builds upon PVT [93] (an attention only pyramid design), by replacing the absolute position embedding in PVT with relative conditional position embedding [113], and incorporating the separable depth-wise convolutions instead of the standard spatial attention, to capture local and global c”
Swin Transformer V2: Scaling Up Capacity and Resolution (2021, background): “Since these works, numerous vision Transformer variants have been proposed to improve the accuracy at relatively small scale [14,21,34,42,63,68,71,75,77,78,82].”
PVT v2: Improved baselines with Pyramid Vision Transformer (2021, methodology): “5% higher than Swin Transformer [21] and Twins [4], while our parameters and FLOPS are fewer.”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, methodology): “Thus, instead of organizing visual tokens into grids, as recent ViT-based applications [16, 23, 44, 81] do, we propose to perform hierarchical grouping of visual tokens into irregular-shaped segments.”
MambaVision: A Hybrid Mamba-Transformer Vision Backbone (2024, neutral): “Models such as Twins [3] and PVT [32] further enhanced efﬁciency through spatially separable self-attention and hierarchical structures with patch embedding.”
MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, methodology): “[20], PoolFormer Block [81], and LSA block in Twins [80].”
Multi-class Token Transformer for Weakly Supervised Semantic Segmentation (2022, methodology): “Although the class token has been removed in a number of recent transformer methods [7, 8, 29], this work will underline its importance for weakly supervised semantic segmentation.”
Uformer: A General U-Shaped Transformer for Image Restoration (2021, background): “Rather than focusing on image classiﬁcation, recent works [10,13,22,67,82] propose a brunch of Transformer-based backbones for more general high-level vision tasks.”
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, background): “While obtaining superior performance, the computational cost of vision transformers is still intensive and scales up quickly as the numbers of patches, self-attention heads and transformer blocks increase.”
Fast Vision Transformers with HiLo Attention (2022, methodology): “More importantly, our LITv2 beats all the compared SoTA ViTs, achieving the best AP with compelling fast inference speed.”
Contextual Transformer Networks for Visual Recognition (2021, background): “[34] proposes twin transformers: Twins-PCPVT that explores conditional positional encodings in pyramid vision transformer and Twins-SVT that interleaves local & global attention with higher throughputs.”
LocalViT: Analyzing Locality in Vision Transformers (2021, positive): “Some works already contributed towards this goal [20], [1], [34], [35], [36], [37].”
Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer (2021, methodology): “For efficient modeling, there are some works apply local/window constraints to self-attention, proposed by [21, 18, 31, 38, 7, 28, 19, 8], reduces the computation cost.”
Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation (2021, methodology): “To cope with the challenge, various ViT backbones that yield multi-scale representations were proposed for semantic segmentation [5, 11, 19, 28, 29, 33, 36].”
TransNeXt: Robust Foveal Visual Perception for Vision Transformers (2023, neutral): “This design enables the model to compare the detailed features perceived by foveal vision with the comprehensive rough outline of the object perceived by peripheral vision in terms of similarity, thereby integrating and calibrating information across the field of view.”
MPViT: Multi-Path Vision Transformer for Dense Prediction (2021, methodology): “We provide a full summary of comparisons on ImageNet-1K classification in Table 8 by adding more recent Vision Transformers including ViL [71], TnT [21], ViTAE [66], HRFormer [69], and Twins [10].”
Stripformer: Strip Transformer for Fast Image Deblurring (2022, methodology): “Method Swin [27] CCNet [23] Twins [28] Ours Params (M) 20 20 20 20 GoPro (PSNR) 32.”