Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
By Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo. 2021
33,967 citations recorded by the source corpus.
This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with Shifted windows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures. The code and models are publicly available at https://github.com/microsoft/Swin-Transformer.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Dacheng Tao
- Quoc V. Le
- Yoshua Bengio
- Trevor Darrell
- Percy Liang
- Sergey Levine
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Yi Yang
- Alan L. Yuille
In Model Architectures
- Jianfeng Gao
- Quoc V. Le
- Alan L. Yuille
- Jiashi Feng
- Jianwei Yang
- Christoph Feichtenhofer
- Lu Yuan
- Shuicheng Yan
Citations in the corpus
- Benchmarking Detection Transfer Learning with Vision Transformers (2021, methodology): “We note that recent work, such as Swin [29] and MViT [12], address the single vs . multi-scale feature map problem by modifying the core ViT architecture (in pre-training) so it is inherently multi-scale.”
- Early Convolutions Help Transformers See Better (2021, background): “Dominant themes include multi-scale networks [14, 17, 25, 45, 50], increasing depth [42], and locality priors [5, 9, 17, 47, 49].”
- VOLO: Vision Outlooker for Visual Recognition (2021, methodology): “%) by Swin Transformer [37], which is pretrained on ImageNet-22k.”
- Towards Robust Vision Transformer (2021, methodology): “Although Swin can also get competitive accuracy, in this work we find such local window self-attention is harmful to the model robustness.”
- Multiview Transformers for Video Recognition (2022, methodology): “In practice, due to the limited amount of training data, applying similar multiscale processing in transformers [11, 21, 44, 74] to reduce complexity of the model has proven to be effective.”
- Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “For image recognition, vision Transform-ers [9, 17, 45, 66, 68, 71, 73, 76] replace CNNs by an image patch tokenizer and a transformer encoder, which have been proved to obtain competitive performance as CNNs. [4, 5, 74] make attempts to apply Transformers on video recognition in a convolution-free…”
- Florence: A New Foundation Model for Computer Vision (2021, methodology): “Our Video CoSwin adapter can borrow the image encoder from CoSwin for the video domain with minimum changes, similar to prior work (Liu et al., 2021b).”
- Achieving Human Parity on Visual Question Answering (2021, methodology): “Swin Transformer [Liu et al., 2021] further builds the model on multi-scale feature maps and uses a shifted windowing scheme on image patches, which makes it compatible with a broad range of vision tasks such as image classification and object detection.”
- Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “On average, training MobileNetV3-Large, ResNet-50, and SwinTransformer-Tiny is 1 .”
- A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “For the visual backbone, we adopt pretrained Swin-T/L [35] by default.”
- Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, methodology): “OurSwin Transformer Tiny models are trained on 16 Nvidia V100 GPUs with a total batch size of 32.”
- Generalized Decoding for Pixel, Image, and Language (2022, methodology): “Model architecture inspection among Swin [50], FocalNet [83] and DaViT [16].”
- ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, background): “Its pre-training setting is the same as UniCL Swin, but external knowledge such as Wiktionary is leveraged in model pre-training.”
- Visual Prompt Tuning (2022, methodology): “We experiment with two Transformer architectures in vision, Vision Transformers (ViT) [19] and Swin Transformers (Swin [52]).”
- Grounded Language-Image Pre-training (2021, methodology): “We first choose the SoTA detection model Dynamic Head (Dy-Head) [10] based on the Swin-Tiny Transformer backbone [39] as the base object detection mode.”
- OneFormer: One Transformer to Rule Universal Image Segmentation (2022, methodology): “(ii) How can the multi-task model better learn inter-task and inter-class differences during the single joint training process? Following the recent success of transformer frameworks [3,12,21,22,27,38,60] in computer vision, we formulate our framework as a transformer-based approach, which can be gu”