Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer

By Zilong Huang, Youcheng Ben, Guozhong Luo, Pei Cheng, Gang Yu, Bin Fu. 2021

214 citations recorded by the source corpus.

Very recently, Window-based Transformers, which computed self-attention within non-overlapping local windows, demonstrated promising results on image classification, semantic segmentation, and object detection. However, less study has been devoted to the cross-window connection which is the key element to improve the representation ability. In this work, we revisit the spatial shuffle as an efficient way to build connections among windows. As a result, we propose a new vision transformer, named Shuffle Transformer, which is highly efficient and easy to implement by modifying two lines of code. Furthermore, the depth-wise convolution is introduced to complement the spatial shuffle for enhancing neighbor-window connections. The proposed architectures achieve excellent performance on a wide range of visual tasks including image-level classification, object detection, and semantic segmentation. Code will be released for reproduction.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Ming-Hsuan Yang
4. Chunhua Shen
5. Xiaogang Wang
6. Yu Qiao
7. Alan L. Yuille
8. Yi Yang

In Electrical and Electronic Engineering

1. Dacheng Tao
2. Jing Zhang
3. Lu Yuan
4. Yu Qiao
5. Furu Wei
6. Bin Xiao
7. Rong Jin
8. Mubarak Shah

Citations in the corpus

• Transformers in Vision: A Survey (2021, methodology): “Shuffle Transformer [95] performs selfattention within a window and has depth-wise convolutions”
• Swin Transformer V2: Scaling Up Capacity and Resolution (2021, background): “Since these works, numerous vision Transformer variants have been proposed to improve the accuracy at relatively small scale [8, 14, 23, 31, 50, 55, 58, 62, 64, 65, 67].”
• Uformer: A General U-Shaped Transformer for Image Restoration (2021, background): “Rather than focusing on image classification, recent works [10, 14, 23, 70, 85] propose a brunch of Transformerbased backbones for more general high-level vision tasks.”
• DaViT: Dual Attention Vision Transformers (2022, background): “Hierarchical designs are widely adopted to transformers [37,56,58,16,75,52,40,69,33,1,31,77,48,52,8,30,67,28,64,32,78] in vision.”
• Activating More Pixels in Image Super-Resolution Transformer (2022, methodology): “A series of Transformer-based methods [13,27,56,51,36,18,12,6,57,60,25,42] have been developed for high-level vision tasks, including image classification [36,13,27,44,49], object detection [34,48,36,4,6], segmentation [55,51,16,2], etc.”
• CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation (2021, background): “2021) propose DeiT that introduces a distillation strategy for transformers to help with ViT training; many other ViT variants (Yuan et al. 2021a; Wang et al. 2021b; Han et al. 2021a; Chen et al. 2021; Ranftl, Bochkovskiy, and Koltun 2021; Liu et al. 2021; Chen, Fan, and Panda 2021; Zhang et al. 202”
• CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers (2022, methodology): “To attain such local-global properties efficiently, we propose a sparse 3D attention model called fused axial attention (FAX), which performs both local window-based attention and sparse global interactions, inspired by [32, 48, 49].”
• Vision Transformers for Single Image Dehazing (2022, methodology): “Subsequently, many modified architectures [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41] have been proposed, and vision Transformer is challenging the dominance of CNNs in high-level vision tasks.”
• TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation (2022, methodology): “For the Feed-Forward Network, we follow [19, 48] to enhance the local connections of Vision Transformer by inserting a depth-wise convolution layer between the two 1×1 convolution layers.”
• Fast Vision Transformers with HiLo Attention (2022, methodology): “In CV, representative efficient self-attention mechanisms includes spatial reduction attention (SRA) [51], local window attention [32, 26] and Twins attention [12].”
• Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer (2021, background): “2021), object detection (Carion et al. 2020; Huang et al. 2021; Liu et al. 2021; Zhu et al. 2020) and segmentation (Duke et al.”
• Vision Transformer Adapter for Dense Predictions (2022, methodology): “From Table 3, we find that our ViT-Adapter achieves comparable accuracy with specific-designed transformers [35,52,75,79].”
• Gold-YOLO: Efficient Object Detector via Gather-and-Distribute Mechanism (2023, methodology): “To establish our Feed-Forward Network, we follow the methodologies presented in [16, 42] for constructing the FFN block.”
• ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias (2021, background): “[80, 28] design sequential stage-wise structures while [47, 33] apply attention within local windows.”
• CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention (2021, methodology): “The competitors are all competitive vision transformers, including DeiT (Touvron et al., 2021), PVT (Wang et al., 2021), T2T-ViT (Yuan et al., 2021), TNT (Han et al., 2021), CViT (Chen et al., 2021a), Twins (Chu et al., 2021), Swin (Liu et al., 2021b), NesT (Zhang et al., 2021b), CvT (Wu et al., 202”
• A Survey on Vision Transformer (2020, methodology): “Shuffle Transformer [105], [63] further utilizes the spatial shuffle operation instead of shifted window partitioning to allow cross-window connections.”