Shunted Self-Attention via Multi-Scale Token Aggregation

By Sucheng Ren, Daquan Zhou, Shengfeng He, Jiashi Feng, Xinchao Wang. 2021

322 citations recorded by the source corpus.

Recent Vision Transformer (ViT) models have demonstrated encouraging results across various computer vision tasks, thanks to its competence in modeling long-range de-pendencies of image patches or tokens via self-attention. These models, however, usually designate the similar receptive fields of each token feature within each layer. Such a constraint inevitably limits the ability of each self-attention layer in capturing multi-scale features, thereby leading to performance degradation in handling images with multiple objects of different scales. To address this issue, we propose a novel and generic strategy, termed shunted self-attention (SSA), that allows ViTs to model the attentions at hybrid scales per attention layer. The key idea of SSA is to inject heterogeneous receptive field sizes into tokens: before computing the self-attention matrix, it selectively merges tokens to represent larger object features while keeping certain tokens to preserve fine-grained features. This novel merging scheme enables the self-attention to learn relationships between objects with different sizes, and simultaneously reduces the token numbers and the computational cost. Extensive experiments across various tasks demonstrate the superiority of SSA. Specifically, the SSA-based transformer achieve 84.0% Top-1 accuracy and out-performs the state-of-the-art Focal Transformer on Ima-geNet with only half of the model size and computation cost, and surpasses Focal Transformer by 1.3 mAP on COCO and 2.9 mIOU on ADE20K under similar parameter and computation cost. Code has been released at https://github.com/OliverRensulShunted-Transformer.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xinchao Wang
2. Jiashi Feng
3. Shuicheng Yan
4. Weihao Yu
5. Dacheng Tao
6. Pan Zhou
7. Daquan Zhou
8. Rynson W. H. Lau

Citations in the corpus

• Learning with Recoverable Forgetting (2022, neutral): “It has been successfully conducted in deep model compression [61], incremental learning [37], continual learning [25,57] and other tasks other than classification [5,53,22,50,36,60,51,55,23].”
• InceptionNeXt: When Inception Meets ConvNeXt (2023, background): “Follow-up works [20, 52, 53, 64, 67, 69, 79] like Swin [39] continually improve model performance, achieving new state-of-the-art on various vision tasks.”
• BiFormer: Vision Transformer with Bi-Level Routing Attention (2023, methodology): “Specifically, in stage i , we use an overlapped patch embedding in the first stage and a patch merging module [25, 37] in the second to fourth stages to reduce the input spatial resolution while increasing the number of channels, followed by N i consecutive BiFormer blocks to transform the features.”
• Factorizing Knowledge in Neural Networks (2022, background): “KD is originally designed for model compression [5,56,32,37,64,51,22], but it has been found to be beneficial in other tasks like adversarial defense [47], domain adaptation [15,44], continual learning [33,68] and amalgamate the knowledge from multiple teachers [39,65,24].”
• MetaFormer Baselines for Vision (2022, methodology): “The power of MetaFormer can also be verified by the recent models adopting MetaFormer as the general architecture but with different attention-based [40, 57, 70], MLP-based [66, 67, 27, 6, 41], convolution-based [15, 19, 79, 56, 77], hybrid [61, 12, 37, 70] or other types of [64, 20] token mixers.”
• DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition (2023, background): “[35], [41], [51], [53], [70], [93], [114] is beneficial for dense prediction tasks such as object detection, instance and semantic segmentation, recent works [21], [23], [32], [38], [42], [54], [58], [66], [77], [78], [95], [104] introduce multi-scale modeling capability by using a pyramid structure”
• Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning (2022, methodology): “Following this direction, Dosovitskiy et al. [11] first employ a transformer in the domain of computer vision and introduce a new architecture paradigm, ViT, which achieves promising results [64, 48].”