UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer

By Haonan Wang, Peng Cao, Osmar R Zaiane. AAAI Conference on Artificial Intelligence, 2021

1,223 citations recorded by the source corpus.

Most recent semantic segmentation methods adopt a U-Net framework with an encoder-decoder architecture. It is still challenging for U-Net with a simple skip connection scheme to model the global multi-scale context: 1) Not each skip connection setting is effective due to the issue of incompatible feature sets of encoder and decoder stage, even some skip connection negatively influence the segmentation performance; 2) The original U-Net is worse than the one without any skip connection on some datasets. Based on our findings, we propose a new segmentation framework, named UCTransNet (with a proposed CTrans module in U-Net), from the channel perspective with attention mechanism. Specifically, the CTrans (Channel Transformer) module is an alternate of the U-Net skip connections, which consists of a sub-module to conduct the multi-scale Channel Cross fusion with Transformer (named CCT) and a sub-module Channel-wise Cross-Attention (named CCA) to guide the fused multi-scale channel-wise information to effectively connect to the decoder features for eliminating the ambiguity. Hence, the proposed connection consisting of the CCT and CCA is able to replace the original skip connection to solve the semantic gaps for an accurate automatic medical image segmentation. The experimental results suggest that our UCTransNet produces more precise segmentation performance and achieves consistent improvements over the state-of-the-art for semantic segmentation across different datasets and conventional architectures involving transformer or U-shaped framework. Code: https://github.com/McGregorWwww/UCTransNet.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/7623a054b85bed188ebd608915ee7a15fc947f7f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

EMCAD: Efficient Multi-Scale Convolutional Attention Decoding for Medical Image Segmentation (2024, background): “Recently, vision transformers have emerged as a formidable force in medical image segmentation, harnessing the ability to capture pixel relationships at global scales [5, 8, 17, 42, 43, 52, 58, 61].”
Medical Image Segmentation Review: The Success of U-Net (2022, methodology): “Lastly, we selected three Transformerutilized U-shaped networks, TransUNet [38], UCTransNet [190] and MISSFormer [49], to demonstrate the Transformer evolution to the medical segmentation field by effectively capturing global contextual information.”
A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation (2024, methodology): “Empowered with its capability of sequence-to-sequence modeling, the Vit demonstrates significantly improved performance in extracting global contextual information as compared to the CNN-based methods, which has inspired many following works [31], [42], [43], [44], [45], [46].”
DCSAU-Net: A Deeper and More Compact Split-Attention U-Net for Medical Image Segmentation (2022, background): “The sufficient feature maps extracted from this layer is able to help to compensate for the spatial information lost during the pooling operations [13].”
U-Shape Transformer for Underwater Image Enhancement (2021, methodology): “To reinforce the network’s attention on the more serious attenuation color channels, inspired by [43], we designed the CMSFFT block to replace the skip connection of the original generator’s encoding-decoding architecture (Fig.”