UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation

By Zongwei Zhou, Mahfuzur R. Siddiquee, Nima Tajbakhsh, Jianming Liang. IEEE Transactions on Medical Imaging, 2019

3,790 citations recorded by the source corpus.

The state-of-the-art models for medical image segmentation are variants of U-Net and fully convolutional networks (FCN). Despite their success, these models have two limitations: (1) their optimal depth is apriori unknown, requiring extensive architecture search or inefficient ensemble of models of varying depths; and (2) their skip connections impose an unnecessarily restrictive fusion scheme, forcing aggregation only at the same-scale feature maps of the encoder and decoder sub-networks. To overcome these two limitations, we propose UNet++, a new neural architecture for semantic and instance segmentation, by (1) alleviating the unknown network depth with an efficient ensemble of U-Nets of varying depths, which partially share an encoder and co-learn simultaneously using deep supervision; (2) redesigning skip connections to aggregate features of varying semantic scales at the decoder sub-networks, leading to a highly flexible feature fusion scheme; and (3) devising a pruning scheme to accelerate the inference speed of UNet++. We have evaluated UNet++ using six different medical image segmentation datasets, covering multiple imaging modalities such as computed tomography (CT), magnetic resonance imaging (MRI), and electron microscopy (EM), and demonstrating that (1) UNet++ consistently outperforms the baseline models for the task of semantic segmentation across different datasets and backbone architectures; (2) UNet++ enhances segmentation quality of varying-size objects—an improvement over the fixed-depth U-Net; (3) Mask RCNN++ (Mask R-CNN with UNet++ design) outperforms the original Mask R-CNN for the task of instance segmentation; and (4) pruned UNet++ models achieve significant speedup while showing only modest performance degradation. Our implementation and pre-trained models are available at https://github.com/MrGiovanni/UNetPlusPlus.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/42b0a8f757e45462e627e57f9af7e9849dcdacdf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation (2021, methodology): “…ETIS mDice mIoU mDice mIoU mDice mIoU mDice mIoU mDice mIoU U-Net [18] 0.818 0.746 0.823 0.750 0.512 0.444 0.710 0.627 0.398 0.335 U-Net++ [33] 0.821 0.743 0.794 0.729 0.483 0.410 0.707 0.624 0.401 0.344 ResUNet++ [13] 0.813 0.793 0.796 0.796 - - - - - - PraNet [8] 0.898 0.840 0.899 0.849…”
VerSe: A Vertebrae Labelling and Segmentation Benchmark for Multi-detector CT Images (2020, methodology): “To perform vertebral body segmentation, a UNet++ architecture using skip connections, multi-class structure, and deep supervision is designed based on work by (Zhou et al., 2019) .”
SNUNet-CD: A Densely Connected Siamese Network for Change Detection of VHR Images (2021, methodology): “Many studies [4], [13]–[15] indicate that shallow layers of the neural network contain ﬁne-grained localization information, while deep layers contain coarse-grained semantic information.”
DoubleU-Net: A Deep Convolutional Neural Network for Medical Image Segmentation (2020, background): “[20] proposed UNet++, a neural network architectures for semantic and instance segmentation tasks.”
Artificial Intelligence for the Metaverse: A Survey (2022, background): “The powerful capability of CNNs in extracting deep visual features at multi-scale image resolutions has been exploited to design advanced segmentation models in [46]–[50].”
AAU-Net: An Adaptive Attention U-Net for Breast Lesions Segmentation in Ultrasound Images (2022, methodology): “In general, the U-net-based variant network (such as U-net++ and U-net3+) achieves better segmentation results than the original U-net, which indicates that the use of skip-connection operations to fuse low-level features in the encoding stage with high-level features in the decoding stage is benefi”
Cross-level Feature Aggregation Network for Polyp Segmentation (2023, methodology): “The results of UNet [10] , UNet++ [10] , SFA [11] , and PraNet [11] are collected from https://github.com/ DengPingFan/PraNet .”
Learning with Limited Annotations: A Survey on Deep Semi-Supervised Learning for Medical Image Segmentation (2022, background): “skip connections [8], incorporating residual/dense convolution blocks [9], [10], attention mechanisms [11], [12], etc.”
Sharp U-Net: Depthwise Convolutional Network for Biomedical Image Segmentation (2021, methodology): “We evaluate the performance of the proposed method against various baselines, including U-Net [10], Wide U-Net [14], TernausNet-16 [15], and U-Net + ResNet-50 [23].”
Mamba-UNet: UNet-Like Pure Visual Mamba for Medical Image Segmentation (2024, neutral): “…U-Net with advanced network blocks techniques such as dense connections [12], residual blocks [10], attention mechanisms [30], depthwise convolutions [11], and atrous convolutions [33,35], resulting in various modified UNet in CT, MRI, Ultrasound medical image segmentation [23,13,15,29,34,36].”
Transformers in Medical Imaging: A Survey (2022, background): “From left to right: Input image, Ground truth label, HIS Hyper (CNN-based) [113], UNet (CNN-based) [114], Attn UNet (CNN-based) [115], UNet++ (CNN-based) [95], and SpecTr (ViT-based) [116].”
TransMorph: Transformer for unsupervised medical image registration (2021, positive): “This limits the U-Net’s ability to perceive semantic information and model long-range relationships between points.”
Deep learning based brain tumor segmentation: a survey (2020, methodology): “In the future, we anticipate the application of network architecture search for searching effective brain tumor segmentation architectures [118], [119], [120], [121].”
Dense Nested Attention Network for Infrared Small Target Detection (2021, background): “segmentation [26], [27], [28], [29] and hybrid attention in”