TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
By Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L. Yuille, Yuyin Zhou. arXiv.org, 2021
5,990 citations recorded by the source corpus.
Medical image segmentation is an essential prerequisite for developing healthcare systems, especially for disease diagnosis and treatment planning. On various medical image segmentation tasks, the u-shaped architecture, also known as U-Net, has become the de-facto standard and achieved tremendous success. However, due to the intrinsic locality of convolution operations, U-Net generally demonstrates limitations in explicitly modeling long-range dependency. Transformers, designed for sequence-to-sequence prediction, have emerged as alternative architectures with innate global self-attention mechanisms, but can result in limited localization abilities due to insufficient low-level details. In this paper, we propose TransUNet, which merits both Transformers and U-Net, as a strong alternative for medical image segmentation. On one hand, the Transformer encodes tokenized image patches from a convolution neural network (CNN) feature map as the input sequence for extracting global contexts. On the other hand, the decoder upsamples the encoded features which are then combined with the high-resolution CNN feature maps to enable precise localization. We argue that Transformers can serve as strong encoders for medical image segmentation tasks, with the combination of U-Net to enhance finer details by recovering localized spatial information. TransUNet achieves superior performances to various competing methods on different medical applications including multi-organ segmentation and cardiac segmentation. Code and models are available at https://github.com/Beckschen/TransUNet.
Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging
View Paper: https://www.semanticscholar.org/paper/24b8a0b02bcb7934967757fc59d273a71ba67e30
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Alan L. Yuille
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Yi Yang
In Radiology, Nuclear Medicine and Imaging
- Vishal M. Patel
- Jeya Maria Jose Valanarasu
- Radu Marculescu
- Md Mostafijur Rahman
- Dong Liu
- Kaiwen Zhang
- Suncheng Xiang
- Jiacheng Ruan
Citations in the corpus
- DCSAU-Net: A Deeper and More Compact Split-Attention U-Net for Medical Image Segmentation (2022, methodology): “…0.955±0.045 0 .876±0.111 0.927±0.131 0.889±0.133 0.817±0.150 UNet3+ [21] 0.957±0.044 0 .889±0.149 0.909±0.135 0.893±0.133 0.825±0.150 TransUNet [ 32] 0.954±0.047 0 .900±0.101 0 .906±0.121 0.895±0.099 0 .821±0.136 LeViT-UNet [45] 0.953±0.049 0 .889±0.150 0 .888±0.147 0.882±0.136 0.808±0.157…”
- Multiscale Vision Transformers (2021, background): “An emerging thread of work aims at applying transformers to vision tasks such as object detection [5], semantic segmentation [121, 105], 3D reconstruction [77], pose estimation [113], generative modeling [17], image retrieval [30], medical image segmentation [16,103,117], point clouds [45], video in”
- Mixed Transformer U-Net for Medical Image Segmentation (2021, background): “Recently, many works try to solve this problem by using Transformer encoder [5, 6, 7].”
- Medical Image Segmentation Review: The Success of U-Net (2022, methodology): “TransUNet[34]suffersfromparametersoverload withinMHSA,whichtreatsallimagetokenizationpositions equally.”
- LViT: Language Meets Vision Transformer in Medical Image Segmentation (2022, positive): “As can be seen from the red boxes in Figure 4, UNet ++ , nnUNet and TransUNet all have more severe mis-segmentation than LViT.”
- UNeXt: MLP-based Rapid Medical Image Segmentation Network (2022, methodology): “TransUNet [6] modi-fies the ViT architecture [10] into an UNet for 2D medical image segmentation.”
- CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation (2021, methodology): “Although the design of TransUNet is interesting and the D e T r a n s L a y e r D e T r a n s L a y e r ...”
- TransAttUnet: Multi-Level Attention-Guided U-Net With Transformer for Medical Image Segmentation (2021, methodology): “Especially, some recent works, such as TransUNet [14] and MedT [15], have tried to incorporate Transformer with CNN-based model to boost the performance of medical image segmentation.”
- Medical Image Segmentation Using Deep Learning: A Survey (2020, background): “Recently, a large number of researches [180] [181] [182] [183] have applied the transformer to medical image segmentation.”
- Channel prior convolutional attention for medical image segmentation (2023, methodology): “For example, TransUNet (Chen et al., 2021b) integrated Transformers to extract contextual information from the feature maps generated by the CNN encoder, thereby enhancing the performance of medical image segmentation.”
- Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis (2021, methodology): “Comparing to prior works [6,47], which are using transformers as secondary encoder, we propose to utilize transformers to embed high-dimensional volumetric medical images, which allow for a more direct encoding of 3D patches and positional embeddings.”
- Transformers in Medical Imaging: A Survey (2022, methodology): “In other work, Lin et al. propose Dual Swin Transformer UNet (DS-TransUNet) [176] to incorporate the advantages of Swin Transformer in U-shaped architecture for medical image segmentation.”
- Transformer-Based Visual Segmentation: A Survey (2023, background): “TransUNet [345] merges transformer and U-Net, where the transformer encodes tokenized image patches to build the global context.”
- Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation (2021, methodology): “In particular, accurate and robust medical image segmentation can play a cornerstone role in computer-aided diagnosis and image-guided clinical surgery [1,2].”
- MedNeXt: Transformer-driven Scaling of ConvNets for Medical Image Segmentation (2023, background): “Transformers [30,7,21] have seen wide-scale adoption in medical image segmentation as either components of hybrid architectures [3,9,33,2,8,31] or standalone techniques [34,25,15] for state-of-the-art performance.”
- UNETR++: Delving Into Efficient and Accurate 3D Medical Image Segmentation (2022, background): “TransUNet [5] combines transformers and the U-Net architecture, where transformers encode the embedded image patches from convolution features and the decoder combines the upsampled encoded features with high-resolution CNN features for localization.”