UNETR: Transformers for 3D Medical Image Segmentation

By Ali Hatamizadeh, Dong Yang, Holger R. Roth, Daguang Xu. IEEE Workshop/Winter Conference on Applications of Computer Vision, 2021

2,828 citations recorded by the source corpus.

Fully Convolutional Neural Networks (FCNNs) with contracting and expanding paths have shown prominence for the majority of medical image segmentation applications since the past decade. In FCNNs, the encoder plays an integral role by learning both global and local features and contextual representations which can be utilized for semantic output prediction by the decoder. Despite their success, the locality of convolutional layers in FCNNs, limits the capability of learning long-range spatial dependencies. Inspired by the recent success of transformers for Natural Language Processing (NLP) in long-range sequence learning, we reformulate the task of volumetric (3D) medical image segmentation as a sequence-to-sequence prediction problem. We introduce a novel architecture, dubbed as UNEt TRansformers (UNETR), that utilizes a transformer as the encoder to learn sequence representations of the input volume and effectively capture the global multi-scale information, while also following the successful "U-shaped" network design for the encoder and decoder. The transformer encoder is directly connected to a decoder via skip connections at different resolutions to compute the final semantic segmentation output. We have validated the performance of our method on the Multi Atlas Labeling Beyond The Cranial Vault (BTCV) dataset for multi-organ segmentation and the Medical Segmentation Decathlon (MSD) dataset for brain tumor and spleen segmentation tasks. Our benchmarks demonstrate new state-of-the-art performance on the BTCV leaderboard.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/7519a1e9e7371df79bd8a21cee871feb0ec597a5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

nnFormer: Volumetric Medical Image Segmentation via a 3D Transformer (2021, methodology): “Moreover, nnFormer is able to surpass the second best method, i.e., UNETR, by large margins in 2 Similar to Synapse, we also follow the evaluation setting of TransUNet.”
Channel prior convolutional attention for medical image segmentation (2023, methodology): “The 3d medical input image is divided into fixed-size blocks by UNETR (Hatamizadeh et al., 2022) and directly fed into a Transformer encoder.”
iSegFormer: Interactive Segmentation via Transformers with Application to 3D Knee MR Images (2021, methodology): “Among these methods, TransUnet [18], Swin-Unet [11], UTNet [17] and UNETR [12] are ”U-shaped” networks that combine transformers and Unets [19] for medical image segmentation.”
AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation (2022, methodology): “For example, UNet with much less parameters surpasses nnFormer, Swin-UNetr over 2 ∼ 3 % mDice and 4 ∼ 5 % mNSD scores on AMOS-CT .”
Semi-Supervised Medical Image Segmentation via Cross Teaching between CNN and Transformer (2021, background): “After that, many variants of vision transformers achieved great success in natural image recognition tasks, like Swin-Transformer (Liu et al., 2021), DieT (Touvron et al., 2021), PVT (Wang et al., 2021b), TiT (Han et al., 2021), etc. Beneﬁting from the great representation capacity of transformers, ”
TransMed: Transformers Advance Multi-Modal Medical Image Classification (2021, methodology): “UNETR [35] utilizes a pure transformer as the encoder to effectively capture the multi-scale information.”
Transformers in Medical Image Analysis: A Review (2022, methodology): “[72], [76], [84], [90], [99], [103], [104], polyp segmentation [79], [87], [101], skin lesion segmentation [79], [87], [91], [101], hip segmentation [79], prostate segmentation [79],”
MISSFormer: An Effective Transformer for 2D Medical Image Segmentation (2022, background): “Hatamizadeh et al. [35] applied transformer to make a powerful encoder for 3d medical image segmentation with CNN-based decoder.”
Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining (2024, positive): “(UNETR [9], Swin-UNETR [8], nnFormer [31]), and the latest Mamba-based segmentation network U-Mamba [18].”
U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation (2024, methodology): “We compared U-Mamba with two CNN-based segmentation networks (nnU-Net [22] and SegResNet [39]) and two Transformer-based networks (UNETR [17] and SwinUNETR [16]), which are widely used in medical image segmentation competitions.”
Mamba-UNet: UNet-Like Pure Visual Mamba for Medical Image Segmentation (2024, methodology): “UNETR combines ViT with UNet for 3D segmentation [9], while Swin-UNet and DCSUnet further explore purely Swin Vision Transformer network blocks with U-Net-based structure [3,28].”
Curriculum Learning: A Survey (2021, methodology): “Following the new trend in computer vision, an emerging area of research in medical imaging is about transformers (Chen et al., 2021a,b; Gao et al., 2021; Hatamizadeh et al., 2021; Korkmaz et al., 2021; Luthra et al., 2021; Ristea et al., 2021).”
SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentation (2024, methodology): “…Dataset Colorectal cancer (CRC) is the third most common cancer worldwide among men and women, the second leading cause RU-Net [19] 64 MSDenseNet [12] 43 MSD [21] 190 Zhang et al. [7] 388 Our CRC-500 500 of death related to cancer, and the primary cause of death in gastrointestinal cancer [4].”
Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images (2022, methodology): “In medical image analysis, UNETR [16] is the ﬁrst methodology that utilizes a ViT as its encoder without relying on a CNN-based feature extractor.”
UNETR++: Delving Into Efficient and Accurate 3D Medical Image Segmentation (2022, methodology): “Hatamizadeh et al. [13] introduce a 3D hybrid model, UNETR, that combines the long-range spatial dependencies of transformers with the CNN’s inductive bias into a “U-shaped” encoder-decoder architecture.”
UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer (2021, background): “Due to the success of Transformers in many computer vision fields, a new paradigm for medical image segmentation has recently evolved(Zheng et al. 2020; Zhang et al. 2021; Gao et al. 2021; Ji et al. 2021; Gao, Zhou, and Metaxas 2021; Zhang, Liu, and Hu 2021; Hatamizadeh et al. 2021).”
HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation (2022, methodology): “UNETR [29] uses a transformer-based encoder to embed input 3D patches and a CNN-based decoder to achieve the final 3D segmentation results.”