nnFormer: Volumetric Medical Image Segmentation via a 3D Transformer

By Hong-Yu Zhou, J. Guo, Yinghao Zhang, Lequan Yu, Liansheng Wang, Yizhou Yu. IEEE Transactions on Image Processing, 2021

731 citations recorded by the source corpus.

Transformer, the model of choice for natural language processing, has drawn scant attention from the medical imaging community. Given the ability to exploit long-term dependencies, transformers are promising to help atypical convolutional neural networks to learn more contextualized visual representations. However, most of recently proposed transformer-based segmentation approaches simply treated transformers as assisted modules to help encode global context into convolutional representations. To address this issue, we introduce nnFormer (i.e., not-another transFormer), a 3D transformer for volumetric medical image segmentation. nnFormer not only exploits the combination of interleaved convolution and self-attention operations, but also introduces local and global volume-based self-attention mechanism to learn volume representations. Moreover, nnFormer proposes to use skip attention to replace the traditional concatenation/summation operations in skip connections in U-Net like architecture. Experiments show that nnFormer significantly outperforms previous transformer-based counterparts by large margins on three public datasets. Compared to nnUNet, the most widely recognized convnet-based 3D medical segmentation model, nnFormer produces significantly lower HD95 and is much more computationally efficient. Furthermore, we show that nnFormer and nnUNet are highly complementary to each other in model ensembling. Codes and models of nnFormer are available at https://git.io/JSf3i.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/bd9ab344da99022cbbbfd3f5c9c82a0b21c60ad9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Transformers in Medical Image Analysis: A Review (2022, methodology): “Transformer-based methods have also been applied to a variety of segmentation tasks, including abdominal multiorgan segmentation [68], [69], [70], [72], [81], [25], [95], [96],”
UNETR++: Delving Into Efficient and Accurate 3D Medical Image Segmentation (2022, methodology): “DSC measures the overlap between the volumetric segmentation predictions and the voxels of the ground truths, it is deﬁned as follows: where, Y and P denote the ground truths and output probabilities for all voxels, respectively.”
U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation (2024, background): “Due to this complementary feature, many studies have explored incorporating Trans-formers into CNNs via hybrid network architectures, such as TransUNet [5,6], UNETR [17], nnFormer [49], and SwinUNETR [16].”
Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining (2024, neutral): “We report the results of nnFormer [31] based on official implementation and the other baseline results were referenced from [18].”