DS-TransUNet: Dual Swin Transformer U-Net for Medical Image Segmentation

By Ailiang Lin, Bingzhi Chen, Jiayu Xu, Zheng Zhang, Guangming Lu, David Zhang. IEEE Transactions on Instrumentation and Measurement, 2021

922 citations recorded by the source corpus.

Automatic medical image segmentation has made great progress owing to powerful deep representation learning. Inspired by the success of self-attention mechanism in transformer, considerable efforts are devoted to designing the robust variants of the encoder–decoder architecture with transformer. However, the patch division used in the existing transformer-based models usually ignores the pixel-level intrinsic structural features inside each patch. In this article, we propose a novel deep medical image segmentation framework called dual swin transformer U-Net (DS-TransUNet), which aims to incorporate the hierarchical swin transformer into both the encoder and the decoder of the standard U-shaped architecture. Our DS-TransUNet benefits from the self-attention computation in swin transformer and the designed dual-scale encoding, which can effectively model the non-local dependencies and multiscale contexts for enhancing the semantic segmentation quality of varying medical images. Unlike many prior transformer-based solutions, the proposed DS-TransUNet adopts a well-established dual-scale encoding mechanism that uses dual-scale encoders based on swin transformer to extract the coarse and fine-grained feature representations of different semantic scales. Meanwhile, a well-designed transformer interactive fusion (TIF) module is proposed to effectively perform multiscale information fusion through the self-attention mechanism. Furthermore, we introduce the swin transformer block into the decoder to further explore the long-range contextual information during the up-sampling process. Extensive experiments across four typical tasks for medical image segmentation demonstrate the effectiveness of DS-TransUNet, and our approach significantly outperforms the state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6feecbba7eacf002edeee797db2704d15ffbda1b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

nnFormer: Volumetric Medical Image Segmentation via a 3D Transformer (2021, methodology): “Inspired by this idea, SwinUNet [18] utilized hierarchical transformer blocks to construct the encoder and decoder within a U-Net like architecture, based on which DS-TransUNet [19] added one more encoder to accept different-sized inputs.”
A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation (2024, methodology): “Empowered with its capability of sequence-to-sequence modeling, the Vit demonstrates significantly improved performance in extracting global contextual information as compared to the CNN-based methods, which has inspired many following works [31], [42], [43], [44], [45], [46].”
Transformers in Medical Image Analysis: A Review (2022, background): “[72], [76], [84], [90], [99], [103], [104], polyp segmentation [79], [87], [101], skin lesion segmentation [79], [87], [91], [101], hip segmentation [79], prostate segmentation [79],”
MALUNet: A Multi-Attention and Light-weight UNet for Skin Lesion Segmentation (2022, methodology): “DS-TransUNet [11] utilizes patches of different scales as input, and two Fig.”
CaraNet: context axial reverse attention network for segmentation of small medical objects (2021, methodology): “We compared CaraNet with six medical image segmentation models, including state-of-the-art models: U-Net [1], UNet++ [2], ResUNet-mod [43], ResUNet++ [3], SFA [44], PraNet [27], CCBANet [50] and DS-TransUNet [51].”
nnFormer: Interleaved Transformer for Volumetric Segmentation (2021, methodology): “Inspired by this idea, Swin-UNet [3] utilized hierarchical transformer blocks to construct the encoder and decoder within a U-Net like architecture, based on which DS-TransUNet [19] added one more encoder to accept diﬀerent-sized inputs.”
Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis (2021, methodology): “Recent works such as Swin UNet [4] and DS-TransUNet [28] utilize the merits of Swin Transformers for 2D segmentation and achieve promising performance.”
HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation (2022, methodology): “Similarly, DS-TransUNet [33] presents a dual-branch Swin Transformer to capture different semantic scale information in the encoder for the task of medical image segmentation.”
UNETR++: Delving Into Efficient and Accurate 3D Medical Image Segmentation (2022, methodology): “Ds-transunet [21] utilizes a dual-scale encoder based on Swin transformer [22] to handle multi-scale inputs and encode local and global feature representations from different semantic scales through self-attention.”
Medical Image Segmentation Review: The Success of U-Net (2022, methodology): “, DS-TransUNet [170] utilize the Swin Transformer in the dual-path Ushaped structure for modeling the multi-scale patch size to lessen the deficiency of Transformers in capturing the local context.”
Swin Transformer Embedding UNet for Remote Sensing Image Semantic Segmentation (2022, methodology): “With the Swin transformer as the backbone, Cao et al. [31] and Lin et al. [32] developed the U-shaped encoder–decoder framework for medical image semantic segmentation.”