TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation

By Yundong Zhang, Huiye Liu, Qiang Hu. International Conference on Medical Image Computing and Computer-Assisted Intervention, 2021

1,457 citations recorded by the source corpus.

Medical image segmentation - the prerequisite of numerous clinical needs - has been significantly prospered by recent advances in convolutional neural networks (CNNs). However, it exhibits general limitations on modeling explicit long-range relation, and existing cures, resorting to building deep encoders along with aggressive downsampling operations, leads to redundant deepened networks and loss of localized details. Hence, the segmentation task awaits a better solution to improve the efficiency of modeling global contexts while maintaining a strong grasp of low-level details. In this paper, we propose a novel parallel-in-branch architecture, TransFuse, to address this challenge. TransFuse combines Transformers and CNNs in a parallel style, where both global dependency and low-level spatial details can be efficiently captured in a much shallower manner. Besides, a novel fusion technique - BiFusion module is created to efficiently fuse the multi-level features from both branches. Extensive experiments demonstrate that TransFuse achieves the newest state-of-the-art results on both 2D and 3D medical image sets including polyp, skin lesion, hip, and prostate segmentation, with significant parameter decrease and inference speed improvement.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/eb20e67ee20cd4122f3812b695b79a3220f2fe4a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation (2021, methodology): “Similar to [2], [27] and [28] use the complementarity of Transformer and CNN to improve the segmentation capability of the model.”
HiFuse: Hierarchical Multi-Scale Feature Fusion Network for Medical Image Classification (2022, background): “Baloch and Krim [2007] proposed a flexible skew-symmetric shape model to learn to capture latent changes within a certain neighborhood. Song et al.Song et al. [2013] proposed a novel texture descriptor that represents texture features by integrating multi-scale Gabor transform and local binarized hi”
nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation (2024, background): “Despite this, many studies continue to claim methodological superiority without benchmarking against any such standardized baseline with a proven high-quality configuration [38,37,34,12,9,11,40].”
DS-TransUNet: Dual Swin Transformer U-Net for Medical Image Segmentation (2021, methodology): “TransFuse [18] is proposed to improve efﬁciency for global context modeling by fusing transformers and CNNs.”
Channel prior convolutional attention for medical image segmentation (2023, methodology): “Another approach, TransFuse (Zhang et al., 2021), employed a parallel combination of Transformers and CNNs to improve the efficiency of capturing global information.”
UNETR++: Delving Into Efficient and Accurate 3D Medical Image Segmentation (2022, methodology): “It is deﬁned as: where, I denotes the number of classes; V denotes the number of voxels; Y v,i and P v,i denote the ground truths and output probabilities at voxel v for class i , respectively.”
nnFormer: Interleaved Transformer for Volumetric Segmentation (2021, methodology): “TransFuse [43] introduced a BiFusion module to fuse features from the shallow convnet-based encoder and transformer-based segmentation network to make final predictions on 2D images.”
VM-UNet: Vision Mamba UNet for Medical Image Segmentation (2024, critical): “Specifically, in the visualizations of the first row, it is observed that non-target areas are often misclassified by other models, such as TransFuse [47] and UTNetV2 [12], leading to inferior segmentation performance; VM-UNet, on the other hand, exhibits greater stability in this aspect.”
Swin Transformer Embedding UNet for Remote Sensing Image Semantic Segmentation (2022, background): “However, TransUNet [20] and TransFuse [60] pointed out that the pure transformer segmentation network produces unsatisfactory results because transformer only focuses on global modeling and lacks positioning capabilities.”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “TransFuse [261] combines transformers and CNNs in a parallel style, where global dependency and low-level spatial details can be efficiently captured jointly.”
Mixed Transformer U-Net for Medical Image Segmentation (2021, background): “Though some of these work achieved satisfying results [7, 9, 10, 11], they usually rely heavily on large-scale pre-training, causing inconvenience to the use of the methods.”
Medical Transformer: Gated Axial-Attention for Medical Image Segmentation (2021, methodology): “TransFuse [28] was recently proposed for polyp segmentation tasks using a parallel CNN branch and transformer branch fused using a BiFusion module.”
Transformers in Medical Image Analysis: A Review (2022, background): “Zhang et al.[79] proposed TransFuse, which also combines Transformers and Unet in a parallel style.”
MALUNet: A Multi-Attention and Light-weight UNet for Skin Lesion Segmentation (2022, methodology): “TransBTS [9], Swin-UNet [10], DS-TransUNet [11], Trans-Fuse [12], UTNet [13] and UTNetV2 [14] have been proposed, further improving segmentation performances.”