Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis

By Yucheng Tang, Dong Yang, Wenqi Li, Holger R. Roth, Bennett A. Landman, Daguang Xu, Vishwesh Nath, Ali Hatamizadeh. Computer Vision and Pattern Recognition, 2021

876 citations recorded by the source corpus.

Vision Transformers (ViT)s have shown great performance in self-supervised learning of global and local representations that can be transferred to downstream applications. Inspired by these results, we introduce a novel self-supervised learning framework with tailored proxy tasks for medical image analysis. Specifically, we propose: (i) a new 3D transformer-based model, dubbed Swin UNEt TRansformers (Swin UNETR), with a hierarchical encoder for self-supervised pretraining; (ii) tailored proxy tasks for learning the underlying pattern of human anatomy. We demonstrate successful pre-training of the proposed model on 5,050 publicly available computed tomography (CT) images from various body organs. The effectiveness of our approach is validated by fine-tuning the pre-trained models on the Beyond the Cranial Vault (BTCV) Segmentation Challenge with 13 abdominal organs and segmentation tasks from the Medical Segmentation Decathlon (MSD) dataset. Our model is currently the state-of-the-art on the public test leaderboards of both MSD11https://decathlon-10.grand-challenge.org/evaluation/challenge/leaderboard/ and BTCV 22https://www.synapse.org/#!Synapse:syn3193805/wiki/217785/ datasets. Code: https://monai.io/research/swin-unetr.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/076a8e778f2e9efb3c2fd45fed534ae9e6035f1b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation (2024, methodology): “The framework is tested by means of three featured architectures ("SegResNet" [28], "DiNTS" [18], "SwinUNETR" [32]).”
Continual Self-Supervised Learning: Towards Universal Multi-Modal Medical Data Representation Learning (2023, neutral): “…a pattern: they primarily focus on a single modality data, such as X-rays [2, 4, 6, 16, 39, 56, 57], computed tomography (CT) scans [16, 19, 41, 50, 52, 53, 56–58], magnetic resonance image (MRI) scans [6, 22, 33, 55], pathological images [23, 44, 51], fundus images [27], multivariate…”
Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images (2022, background): “In addition, ViTs have achieved success in eﬀective learning of pretext tasks for self-supervised pre-training in various applications [9,8,36].”
Semi-Supervised Medical Image Segmentation via Cross Teaching between CNN and Transformer (2021, methodology): “…2021b), deep co-training (Qiao et al., 2018; Zhou et al., 2019), deep adversarial learning (Zhang et al., 2017; Hu et al., 2020), few-shot learning (Tang et al., 2021a), mean teacher and its extensions (Tarvainen and Valpola, 2017; Yu et al., 2019; Li et al., 2020; Reiß et al., 2021; You et al.,…”
Transformers in Medical Imaging: A Survey (2022, methodology): “UNETR achieves impressive performance on BTCV [203] and MSD [171] segmentation datasets as shown in Fig.”
Denoising diffusion probabilistic models for 3D medical image generation (2023, positive): “Since the diffusion model outputs the synthetic images without any ground truth labels, we used the synthetic images to pre-train a Swin UNETR 23 in a self-supervised fashion.”
An OpenMind for 3D Medical Vision Self-supervised Learning (2024, unknown): “Some methods use CNNs for pre-training [15, 17, 21], some ViT-like Transformers [18, 19], others Swin Transformers [22, 23] and even others hybrid architectures [16, 24]. iii) Insufficient downstream dataset choices, potentially providing unreliable results.”