Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images

By Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R. Roth, Daguang Xu. BrainLes@MICCAI, 2022

2,376 citations recorded by the source corpus.

Semantic segmentation of brain tumors is a fundamental medical image analysis task involving multiple MRI imaging modalities that can assist clinicians in diagnosing the patient and successively studying the progression of the malignant entity. In recent years, Fully Convolutional Neural Networks (FCNNs) approaches have become the de facto standard for 3D medical image segmentation. The popular"U-shaped"network architecture has achieved state-of-the-art performance benchmarks on different 2D and 3D semantic segmentation tasks and across various imaging modalities. However, due to the limited kernel size of convolution layers in FCNNs, their performance of modeling long-range information is sub-optimal, and this can lead to deficiencies in the segmentation of tumors with variable sizes. On the other hand, transformer models have demonstrated excellent capabilities in capturing such long-range information in multiple domains, including natural language processing and computer vision. Inspired by the success of vision transformers and their variants, we propose a novel segmentation model termed Swin UNEt TRansformers (Swin UNETR). Specifically, the task of 3D brain tumor semantic segmentation is reformulated as a sequence to sequence prediction problem wherein multi-modal input data is projected into a 1D sequence of embedding and used as an input to a hierarchical Swin transformer as the encoder. The swin transformer encoder extracts features at five different resolutions by utilizing shifted windows for computing self-attention and is connected to an FCNN-based decoder at each resolution via skip connections. We have participated in BraTS 2021 segmentation challenge, and our proposed model ranks among the top-performing approaches in the validation phase. Code: https://monai.io/research/swin-unetr

Subfields: Computer Vision and Pattern Recognition, Neurology

View Paper: https://www.semanticscholar.org/paper/f7410f535bda5b5a9888512fb954193245c1d0b2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Neurology

Citations in the corpus

Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining (2024, positive): “For medical image segmentation, U-Mamba [18] and SegMamba [29] proposed a task-specific architecture with the Mamba block based on nnU-Net [11] and Swin-UNETR [8], respectively.”
Comparison of Vision Transformers and Convolutional Neural Networks in Medical Image Analysis: A Systematic Review (2024, neutral): “Hatamizadeh et al. proposed the Swin-UNEt TRansformer (Swin UNETR), which uses Swin transformers as encoders in a U-shaped network connected to a multiresolution CNN-based decoder via skip links [90].”
Transformers in Medical Imaging: A Survey (2022, methodology): “[145] propose Swin UNet based architecture, Swin UNETR, that consists of Swin transformer as the encoder and a CNN-based decoder.”
AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation (2022, methodology): “For example, UNet with much less parameters surpasses nnFormer, Swin-UNetr over 2 ∼ 3 % mDice and 4 ∼ 5 % mNSD scores on AMOS-CT .”
Transformers in Medical Image Analysis: A Review (2022, background): “mour/tissue segmentation [87], [93], [105], [112], [123], [128], [129], [130], [131], [132], [122], [113], [91], [133], [134], polyp segmentation [96], [108], [125], [135], [136], liver and hep-”
MedNeXt: Transformer-driven Scaling of ConvNets for Medical Image Segmentation (2023, methodology): “Transformers [30,7,21] have seen wide-scale adoption in medical image segmentation as either components of hybrid architectures [3,9,33,2,8,31] or standalone techniques [34,25,15] for state-of-the-art performance.”
MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer (2023, methodology): “…in the community, including the CNN-based method nnUNet(Isensee et al. 2021), the transformer-based methods TransUNet(Chen et al. 2021), UNetr(Hatamizadeh et al. 2022), Swin-UNetr(Jiang et al. 2022) and the diffusion based method EnsDiff (Wolleb et al. 2021), SegDiff(Amit et al. 2021),…”
Medical Image Segmentation Review: The Success of U-Net (2022, methodology): “UNETR [116] is a 3D segmentation network that directly utilizes volumetric data incorporating ViT solely at the encoder stage to capture global multi-scale contextual information in a 3D volumetric style which is usually of paramount importance in medical image segmentation domain.”
SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentation (2024, methodology): “Following many previous studies [12,6,7], we utilize a CNN-based decoder and a skip connections for predicting the segmentation results.”
U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation (2024, methodology): “For medical image segmentation, U-Mamba [56] and SegMamba [88] have proposed task-specific architectures with Mamba blocks respectively based on nn-UNet [32] and Swin UNETR [22], respectively, achieving promising results in various visual tasks and demonstrating the potential of SSMs in vision.”
U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation (2024, methodology): “We compared U-Mamba with two CNN-based segmentation networks (nnU-Net [22] and SegResNet [39]) and two Transformer-based networks (UNETR [17] and SwinUNETR [16]), which are widely used in medical image segmentation competitions.”
CellViT: Vision Transformers for Precise Cell Segmentation and Classification (2023, methodology): “Even though many other adaptations of the U-Net structure for Vision Transformers have been proposed (e.g., SwinUNETR [56]), it was important for us to choose a network structure that incorporates the original ViT structure by Dosovitskiy et al. [50] without modifications such that we can make use…”
nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation (2024, background): “Despite this, numerous studies claim methodological superiority while at least 50% of the benchmark is made up by either BTCV [37,11,9] or BraTS [14,34,31,28,16].”
nnFormer: Volumetric Medical Image Segmentation via a 3D Transformer (2021, methodology): “As we can see, the best performing methods are Swin-UNETR [35], [36] and TransUNet [11].”
UNETR++: Delving Into Efficient and Accurate 3D Medical Image Segmentation (2022, methodology): “%) nnUNet [16] 74.31 SwinUNETR [12] 75.55 nnFormer [36] 77.95 UNETR [13] 73.29 UNETR++ 80.68 Lungs Dataset: We evaluate UNETR++ and other SOTA models on the lung cancer segmentation task.”