CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation

By Yutong Xie, Jianpeng Zhang, Chunhua Shen, Yong Xia. International Conference on Medical Image Computing and Computer-Assisted Intervention, 2021

715 citations recorded by the source corpus.

Convolutional neural networks (CNNs) have been the de facto standard for nowadays 3D medical image segmentation. The convolutional operations used in these networks, however, inevitably have limitations in modeling the long-range dependency due to their inductive bias of locality and weight sharing. Although Transformer was born to address this issue, it suffers from extreme computational and spatial complexities in processing high-resolution 3D feature maps. In this paper, we propose a novel framework that efficiently bridges a {\bf Co}nvolutional neural network and a {\bf Tr}ansformer {\bf (CoTr)} for accurate 3D medical image segmentation. Under this framework, the CNN is constructed to extract feature representations and an efficient deformable Transformer (DeTrans) is built to model the long-range dependency on the extracted feature maps. Different from the vanilla Transformer which treats all image positions equally, our DeTrans pays attention only to a small set of key positions by introducing the deformable self-attention mechanism. Thus, the computational and spatial complexities of DeTrans have been greatly reduced, making it possible to process the multi-scale and high-resolution feature maps, which are usually of paramount importance for image segmentation. We conduct an extensive evaluation on the Multi-Atlas Labeling Beyond the Cranial Vault (BCV) dataset that covers 11 major human organs. The results indicate that our CoTr leads to a substantial performance improvement over other CNN-based, transformer-based, and hybrid methods on the 3D multi-organ segmentation task. Code is available at \def\UrlFont{\rm\small\ttfamily} \url{https://github.com/YtongXie/CoTr}

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8356d155d730e374f4db6dfd03d19a7b66c348a8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images (2022, methodology): “Other approaches [40,39] have attempted to leverage the power of ViTs as a stand-alone block in their architectures which otherwise consist of CNN-based components.”
Medical Image Segmentation Review: The Success of U-Net (2022, methodology): “CoTr[37]isaU-Net-likestructurewithCNN-based3Dresid-ualblocksforencoderanddecoderpathswiththeamalgamation ofDeformableTransformer(DeTrans)formulti-scalefusion, besidestheconventionalskipconnectionsfromtheencoder tothedecoderforbetterlocalizationinformationandfaster convergence.”
Learning with Limited Annotations: A Survey on Deep Semi-Supervised Learning for Medical Image Segmentation (2022, methodology): “Inspired by recent success of transformer architectures in the field of natural language processing, many transformer-based methods have been proposed and applied for medical image segmentation [11, 13].”
nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation (2024, methodology): “Adaptations of these cutting-edge designs to the medical imaging domain have emerged, with claims of superior performance over the conventional CNN-based U-Net [32,11,15,17,41,38,26,9,12,37,34].”
nnFormer: Interleaved Transformer for Volumetric Segmentation (2021, neutral): “[39] adopted transformers to capture long-term dependencies of multi-scale convolutional features from different layers of convnets.”
AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation (2022, methodology): “In this group of work [28, 6, 7, 24], the features patch or image patch are treated as tokens, which are used to conduct efficient non-local context modeling among arbitrary positions, making them achieve SOTA performance on the most popular dataset [12].”
UNETR: Transformers for 3D Medical Image Segmentation (2021, methodology): “For comparison, other transformer-based methods such as CoTr [47], TransUNet [7] and SETR [52] have 46.”
Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation (2021, methodology): “Currently, various combinations of Transformer with CNN are applied in multi-modal brain tumor segmentation [29] and 3D medical image segmentation [1,30].”
Transformers in Medical Image Analysis: A Review (2022, methodology): “Different from TransUNet, which only uses Transformer to process the low-resolution feature maps learned from the previous layer, Xie et al.[25] proposed a deformable Transformer (DeTrans) to process the multi-scale and high-resolution feature maps.”
MISSFormer: An Effective Medical Image Segmentation Transformer (2021, methodology): “Although their superb performance and prevalence, the CNN-based methods suffer from a limitation in modeling the long-range dependencies because of the locality of convolution operation(Cao et al. 2021; Xie et al. 2021b), and they failed to achieve the goal of precise medical image analysis.”
UNETR++: Delving Into Efficient and Accurate 3D Medical Image Segmentation (2022, background): “It is deﬁned as: where, I denotes the number of classes; V denotes the number of voxels; Y v,i and P v,i denote the ground truths and output probabilities at voxel v for class i , respectively.”
TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up (2021, background): “Other works introduce the pyramid/hierarchical structure to transformer [52, 53, 54] or combine it with convolutional layers [55, 56].”