CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows

By Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, Baining Guo. 2021

1,394 citations recorded by the source corpus.

We present CSWin Transformer, an efficient and effective Transformer-based backbone for general-purpose vision tasks. A challenging issue in Transformer design is that global self-attention is very expensive to compute whereas local self-attention often limits the field of interactions of each token. To address this issue, we develop the Cross-Shaped Window self-attention mechanism for computing self-attention in the horizontal and vertical stripes in parallel that form a cross-shaped window, with each stripe obtained by splitting the input feature into stripes of equal width. We provide a mathematical analysis of the effect of the stripe width and vary the stripe width for different layers of the Transformer network which achieves strong modeling capability while limiting the computation cost. We also introduce Locally-enhanced Positional Encoding (LePE), which handles the local positional information better than existing encoding schemes. LePE naturally supports arbitrary input resolutions, and is thus especially effective and friendly for downstream tasks. Incorporated with these designs and a hierarchical structure, CSWin Transformer demonstrates competitive performance on common vision tasks. Specifically, it achieves 85.4% Top-1 accuracy on ImageNet-1K without any extra training data or label, 53.9 box AP and 46.4 mask AP on the COCO detection task, and 52.2 mIOU on the ADE20K semantic segmentation task, surpassing previous state-of-the-art Swin Transformer backbone by +1.2, +2.0, +1.4, and +2.0 respectively under the similar FLOPs setting. By further pretraining on the larger dataset ImageNet-21K, we achieve 87.5% Top-1 accuracy on ImageNet-1K and high segmentation performance on ADE20K with 55.7 mIoU. 11Code and pretrain model is available at https://github.com/microsoft/CSWin-Transformer

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

Florence: A New Foundation Model for Computer Vision (2021, methodology): “For the image encoder, we chose hierarchical Vision Transformers (e.g. , Swin (Liu et al., 2021a), CvT (Wu et al., 2021), Vision Longformer (Zhang et al., 2021a), Focal Transformer (Yang et al., 2021), and CSwin (Dong et al., 2021)).”
Transformers in Vision: A Survey (2021, methodology): “axial-attention [55, 83], low-rank projection attention [191, 207, 224], kernelizable attention [41, 156], and similarity-clustering-based methods [104, 192].”
Swin Transformer V2: Scaling Up Capacity and Resolution (2021, background): “Since these works, numerous vision Transformer variants have been proposed to improve the accuracy at relatively small scale [14,21,34,42,63,68,71,75,77,78,82].”
DeiT III: Revenge of the ViT (2022, neutral): “Table 11: ADE20k semantic segmentation performance using UperNet [59] (in comparable settings [11, 16, 31]).”
Vision Transformers for Single Image Dehazing (2022, methodology): “Subsequently, many modified architectures [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41] have been proposed, and vision Transformer is challenging the dominance of CNNs in high-level vision tasks.”
A Survey on Vision Transformer (2020, background): “Swin Transformers [148], [54] performs local attention within a window and introduces a shifted window partitioning approach for cross-window connections.”
Token Merging: Your ViT But Faster (2022, methodology): “, 2019), and some attempt to infuse domain-specific modules (Mehta & Rastegari, 2021; Graham et al., 2021; Liu et al., 2021; 2022b; Dong et al., 2022).”
Mobile-Former: Bridging MobileNet and Transformer (2021, methodology): “For example, Swin [20] presents shifted window approach for computing self-attention within a local window and CSWin [7] further improves it by introducing cross-shaped window self-attention.”
Rep ViT: Revisiting Mobile CNN From ViT Perspective (2023, background): “Building on the competitive performance of ViTs, subsequent works have sought to incorporate spatial inductive biases to enhance their stability and performance [8,17], design more efficient self-attention operations by regularizing the pattern of attention mechanisms [13, 69], and adapt ViTs to a d”
Transformers in Remote Sensing: A Survey (2022, methodology): “the transformers design such as, reduced computational overhead [167], efficient hybrid CNN-transformers backbones [168] and unified architectures for image and video classification [169].”
Fast Vision Transformers with HiLo Attention (2022, background): “In Figure 6, we show that HiLo is consistently faster than many attention mechanisms [51, 32, 1, 50, 11, 17, 55, 60, 41, 44, 20, 18] on both CPUs and GPUs.”
Residual Mixture of Experts (2022, background): “Triggered by ViT, a series of follow-up works [8, 9, 14, 17, 21, 25, 30, 38, 49, 50, 52, 54, 57, 59, 60] continuously refresh the image classification records and push the potential of vision transformers in computer vision to a new height.”
On the Integration of Self-Attention and Convolution (2021, methodology): “We practically implement ACmix on 4 baseline models, including ResNet [19], SAN [53], PVT [43] and Swin-Transformer [31].”
MambaOut: Do We Really Need Mamba for Vision?* (2024, neutral): “…under the spirit of RNN. Neverthe-less, their experiments show that the SSM based models for vision, in reality, lead to underwhelming performance compared with state-of-the-art convolutional [23, 31, 39, 54, 58, 70, 86, 98, 100, 101] and attention-based models [19, 24, 51, 76, 80, 81, 100, 103].”
Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation (2021, methodology): “We compare the performance and efﬁciency of our HRViT with SoTA ViT backbones, i.e., Swin [19], Twins [5], MiT [33], and CSWin [11].”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, neutral): “Similarly, a METER [56] head is used for vision language representation, and CSwin [57] is used for video-based understanding.”