UTNet: A Hybrid Transformer Architecture for Medical Image Segmentation

By Yunhe Gao, Mu Zhou, Dimitris N. Metaxas. International Conference on Medical Image Computing and Computer-Assisted Intervention, 2021

631 citations recorded by the source corpus.

Transformer architecture has emerged to be successful in a number of natural language processing tasks. However, its applications to medical vision remain largely unexplored. In this study, we present UTNet, a simple yet powerful hybrid Transformer architecture that integrates self-attention into a convolutional neural network for enhancing medical image segmentation. UTNet applies self-attention modules in both encoder and decoder for capturing long-range dependency at different scales with minimal overhead. To this end, we propose an efficient self-attention mechanism along with relative position encoding that reduces the complexity of self-attention operation significantly from $O(n^2)$ to approximate $O(n)$. A new self-attention decoder is also proposed to recover fine-grained details from the skipped connections in the encoder. Our approach addresses the dilemma that Transformer requires huge amounts of data to learn vision inductive bias. Our hybrid layer design allows the initialization of Transformer into convolutional networks without a need of pre-training. We have evaluated UTNet on the multi-label, multi-vendor cardiac magnetic resonance imaging cohort. UTNet demonstrates superior segmentation performance and robustness against the state-of-the-art approaches, holding the promise to generalize well on other medical image segmentations.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/a2a0fc5014423d6283c80511b9272d941109ece2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

MALUNet: A Multi-Attention and Light-weight UNet for Skin Lesion Segmentation (2022, methodology): “TransBTS [9], Swin-UNet [10], DS-TransUNet [11], TransFuse [12], UTNet [13] and UTNetV2 [14] have been proposed, further improving segmentation performances.”
nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation (2024, methodology): “Adaptations of these cutting-edge designs to the medical imaging domain have emerged, with claims of superior performance over the conventional CNN-based U-Net [32,11,15,17,41,38,26,9,12,37,34].”
UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer (2021, background): “Due to the success of Transformers in many computer vision fields, a new paradigm for medical image segmentation has recently evolved(Zheng et al. 2020; Zhang et al. 2021; Gao et al. 2021; Ji et al. 2021; Gao, Zhou, and Metaxas 2021; Zhang, Liu, and Hu 2021; Hatamizadeh et al. 2021).”
Transformers in Medical Imaging: A Survey (2022, neutral): “Figure sources: (a) [34], (b) [35], (c) [36], (d) [37], (e) [38], (f) [39].”
Medical Image Segmentation Using Deep Learning: A Survey (2020, background): “Recently, a large number of researches [180] [181] [182] [183] have applied the transformer to medical image segmentation.”
iSegFormer: Interactive Segmentation via Transformers with Application to 3D Knee MR Images (2021, methodology): “Among these methods, TransUnet [18], Swin-Unet [11], UTNet [17] and UNETR [12] are ”U-shaped” networks that combine transformers and Unets [19] for medical image segmentation.”
Curriculum Learning: A Survey (2021, background): “medical imaging is about transformers (Chen et al. 2021a, b; Gao et al. 2021; Hatamizadeh et al. 2021; Korkmaz et al. 2021; Luthra et al. 2021; Ristea et al. 2021).”
Transformers in Medical Image Analysis: A Review (2022, background): “Gao et al.[74] presented a UTNet that applies self-attention”