Self-Supervised Learning with Swin Transformers

By Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, Han Hu. 2021

209 citations recorded by the source corpus.

We are witnessing a modeling shift from CNN to Transformers in computer vision. In this work, we present a self-supervised learning approach called MoBY, with Vision Transformers as its backbone architecture. The approach basically has no new inventions, which is combined from MoCo v2 and BYOL and tuned to achieve reasonably high accuracy on ImageNet-1K linear evaluation: 72.8% and 75.0% top-1 accuracy using DeiT-S and Swin-T, respectively, by 300-epoch training. The performance is slightly better than recent works of MoCo v3 and DINO which adopt DeiT as the backbone, but with much lighter tricks. More importantly, the general-purpose Swin Transformer backbone enables us to also evaluate the learnt representations on downstream tasks such as object detection and semantic segmentation, in contrast to a few recent approaches built on ViT/DeiT which only report linear evaluation results on ImageNet-1K due to ViT/DeiT not tamed for these dense prediction tasks. We hope our results can facilitate more comprehensive evaluation of self-supervised learning methods designed for Transformer architectures. Our code and models are available at https://github.com/SwinTransformer/Transformer-SSL, which will be continually enriched.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

BEiT: BERT Pre-Training of Image Transformers (2021, methodology): “Apart from masked auto-encoding, other mainstream research works use contrastive learning (Chen et al., 2021; Xie et al., 2021), and self-distillation (Caron et al.”
Transformers in Vision: A Survey (2021, background): “Xie et al.[224] combine MoCo v2 [78] and”
XCiT: Cross-Covariance Image Transformers (2021, methodology): “SSL Method Model #params FLOPs Linear k-NN MoBY [77] Swin-T [44] 29M 4.”
A Survey of Visual Transformers (2021, methodology): “Discriminative Representation MoCo v3 [64], DINO [65], MoBY [66]”
DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting (2021, methodology): “To alleviate the heavy annotation cost in previous supervised pre-training settings, a number of self-supervised pre-training approaches have been proposed for dense prediction [3, 43, 46, 47].”
Efficient Self-supervised Vision Transformers for Representation Learning (2021, background): “MoBY [64] is a con-current work that investigates multi-stage ViT in SSL.”
Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling (2021, background): “image tasks [2, 6, 9, 45, 55, 66], and the capability of diminishing the inductive biases is also justified by enabling more parameters, more data [9], and longer training schedules.”
SODA10M: Towards Large-Scale Object Detection Benchmark for Autonomous Driving (2021, unknown): “These results inspire us when pre-training a transformer-based model under a self-supervised scheme, we need to develop some specific algorithms based on its special structure, such as DeiT [52] and Swin-SSL [60].”
Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis (2021, background): “As opposed to convolutional neural networks (CNNs) with limited receptive fields, ViTs encode visual representations from a sequence of patches and leverage self-attention blocks for modeling long-range global information [38].”
iBOT: Image BERT Pre-Training with Online Tokenizer (2021, methodology): “To compare, we include the results of supervised Swin-T (Liu et al., 2021b) which shares approximate parameter numbers with ViT-S/16 and its self-supervised counterpart MoBY (Xie et al., 2021a) in Tab.”
Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation (2022, background): “However, when it comes to vision Transformer backbones [10, 30], the fine-tuning performance becomes inferior to others [43, 26], which may have hindered its appeal for now.”
Mamba-UNet: UNet-Like Pure Visual Mamba for Medical Image Segmentation (2024, background): “Techniques like shift windows have further tailored ViT, resulting in Swin-Transformer [18], which enhances their applicability in dense prediction tasks in computer vision, such as image segmentation, and detection [19,31,17].”
Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs (2022, background): “been greatly challenged by Vision Transformers (ViTs) [34, 59, 86, 94], which have shown leading performances on many visual tasks – not only image classification [34, 104] and representation learning [4, 9, 16, 100], but also many downstream tasks such as object detection [24, 59], semantic segment”
Patch-level Representation Learning for Self-supervised Vision Transformers (2022, methodology): “We consider a variety of existing SSL methods developed for ResNet [20] and ViT [35] architectures: (a) self-supervised ResNets: MoCo-v2 [7], SwAV [3], DenseCL [39], ReSim [43], and DetCo [44]; and (b) selfsupervised ViTs: DINO [4], MoCo-v3 [9], and MoBY [46].”
BEVT: BERT Pretraining of Video Transformers (2021, methodology): “(2) Image CL: pretraining the image Swin Transformer on Imagenet-1K with a self-supervised contrastive learning method [64].”
A Survey on Vision Transformer (2020, methodology): “Contrastive learning has been applied on vision transformer for unsupervised pretraining [32], [247], [126].”