Swin Transformer V2: Scaling Up Capacity and Resolution

By Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong and 2 more. 2021

2,875 citations recorded by the source corpus.

We present techniques for scaling Swin Transformer [35] up to 3 billion parameters and making it capable of training with images of up to 1,536x1,536 resolution. By scaling up capacity and resolution, Swin Transformer sets new records on four representative vision benchmarks: 84.0% top-1 accuracy on ImageNet- V2 image classification, 63.1 / 54.4 box / mask mAP on COCO object detection, 59.9 mIoU on ADE20K semantic segmentation, and 86.8% top-1 accuracy on Kinetics-400 video action classification. We tackle issues of training instability, and study how to effectively transfer models pre-trained at low resolutions to higher resolution ones. To this aim, several novel technologies are proposed: 1) a residual post normalization technique and a scaled cosine attention approach to improve the stability of large vision models; 2) a log-spaced continuous position bias technique to effectively transfer models pre-trained at low-resolution images and windows to their higher-resolution counterparts. In addition, we share our crucial implementation details that lead to significant savings of GPU memory consumption and thus make it feasi-ble to train large vision models with regular GPUs. Using these techniques and self-supervised pre-training, we suc-cessfully train a strong 3 billion Swin Transformer model and effectively transfer it to various vision tasks involving high-resolution images or windows, achieving the state-of-the-art accuracy on a variety of benchmarks. Code is avail-able at https://github.com/microsoft/Swin-Transformer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

A Review of Sparse Expert Models in Deep Learning (2022, methodology): “their system, they add MoE layers to the Swin Tranformer V2 architecture (Liu et al., 2022) for image classification and object detection.”
VOLO: Vision Outlooker for Visual Recognition (2021, neutral): “Recently, some works [14], [59], [60], [61], [62] aim”
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “We consider various families of models as teachers including ResNets (34–152 and type d variants) [21], ConvNeXt family pretrained on the ImageNet-22K and fine-tuned on ImageNet-1K [39], DeiT-3 pretrained on the ImageNet-21K and fine-tuned on ImageNet-1K, IG-ResNext pretrained on the Instagram datas”
OneFormer: One Transformer to Rule Universal Image Segmentation (2022, neutral): “3 UPerNet‡‡ [41] SwinV2-G†† [29] >3B — — 640×640 80k — — 59.”
Exploring Plain Vision Transformer Backbones for Object Detection (2022, methodology): “For a fairer comparison, here we make two changes following our competitors: we adopt soft-nms [2] as is used by all competitors [39,31,33,38] in this table and increase the input size (from 1024 to 1280) following [33,38].”
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks (2022, background): “Model Extra Data Image Size ImageNet With extra private image-tag data SwinV2-G [LHL21] IN-22K-ext-70M 640(2) 90.”
MambaVision: A Hybrid Mamba-Transformer Vision Backbone (2024, neutral): “In addition, our framework allows for computing the attention in a windowed manner similar to previous efforts [21, 22] (see the supplementary materials for window size ablation study).”
RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation Based on Visual Foundation Model (2023, methodology): “In terms of architecture, they have begun adopting the Transformer for the visual domain, investigating large models such as ViT-G [57], ViT-22B [58], and Swin Transformer v2 [59].”
How Artificial Intelligence Is Shaping Medical Imaging Technology: A Survey of Innovations and Applications (2023, background): “…which were further refined in SWIN v2 [43]. differences, when compared with the window in self-attention a limited the entire different reduce the improve the original ViT, allow a more efficient and scalable architecture, which were further refined in SWIN v2 [43]. refined in SWIN v2 [43].”
Masked Feature Prediction for Self-Supervised Visual Pre-Training (2021, result): “%) is already similar to recent 86.5% Florence [95] and 86.8% SwinV2-G [58].”
VMamba: Visual State Space Model (2024, methodology): “Our training framework is constructed using the MMDetection library [3], and we adhere to the hyper-parameters used in Swin [31] with the Mask-RCNN detector.”
Token Merging: Your ViT But Faster (2022, methodology): “5, we show the results of applying our method off-the-shelf and during MAE fine-tuning using ViT-L from Spatiotemporal MAE compared to the relevant state-of-the-art on Kinetics-400 classification: Swin (Liu et al., 2022c) pretrained on ImageNet-21k, MViTv2 (Li et al., 2022) pretrained with MaskFeats”
ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth (2023, background): “Another type of transformer with good performance is the hierarchical transformer Swin2 [27] based on shifted windows.”
Rethinking Vision Transformers for MobileNet Size and Speed (2022, background): “The promising performance of Vision Transformers (ViTs) [18] has inspired many follow-up works to further refine the model architecture and improve training strategies, leading to superior results on most computer vision benchmarks, such as classification [7, 47, 49, 52], segmentation [5, 13, 77], d”
Decomposing NeRF for Editing via Feature Field Distillation (2022, methodology): “…left to the table” 2020, Liu et al., 2021a, Azuma et al., 2022], but left for future work. view-dependent query like referring expressions (e.g., “the chair left to the table”) [ 2020, Liu et al., 2021a, Azuma et al., 2022], but left for future work. for minimizing the difference between I,r ) .”
TransNeXt: Robust Foveal Visual Perception for Vision Transformers (2023, neutral): “Linear complexity mode for inference : We observe that in Equations 15 and 16, if we consistently set H p and W p as fixed values independent of the input size, the computational complexity of both pixel-focused attention and aggregated attention grows linearly with the length of the input sequence.”