Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network

By Zizhao Zhang, Lin F. Yang, Yefeng Zheng. 2018

420 citations recorded by the source corpus.

Synthesized medical images have several important applications, e.g., as an intermedium in cross-modality image registration and as supplementary training samples to boost the generalization capability of a classifier. Especially, synthesized computed tomography (CT) data can provide X-ray attenuation map for radiation therapy planning. In this work, we propose a generic cross-modality synthesis approach with the following targets: 1) synthesizing realistic looking 3D images using unpaired training data, 2) ensuring consistent anatomical structures, which could be changed by geometric distortion in cross-modality synthesis and 3) improving volume segmentation by using synthetic data for modalities with limited training samples. We show that these goals can be achieved with an end-to-end 3D convolutional neural network (CNN) composed of mutually-beneficial generators and segmentors for image synthesis and segmentation tasks. The generators are trained with an adversarial loss, a cycle-consistency loss, and also a shape-consistency loss, which is supervised by segmentors, to reduce the geometric distortion. From the segmentation view, the segmentors are boosted by synthetic data from generators in an online manner. Generators and segmentors prompt each other alternatively in an end-to-end training fashion. With extensive experiments on a dataset including a total of 4,496 CT and magnetic resonance imaging (MRI) cardiovascular volumes, we show both tasks are beneficial to each other and coupling these two tasks results in better performance than solving them exclusively.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Generative Models, Medical Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Generative Models

In Medical Imaging

Citations in the corpus

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017, background): “Compared with other deep generative models, Generative Adversarial Networks (GANs) [6] have shown great performance for generating sharper samples [17, 3, 23, 13, 10, 35, 24, 34, 39, 40].”
Photographic Text-to-Image Synthesis with a Hierarchically-Nested Adversarial Network (2018, methodology): “There are substantial existing methods investigating the better usage of GANs for different applications, such as image synthesis [32, 38], (unpaired) pixelto-pixel translation [14, 51], medical applications [4, 49], etc [20, 12, 52, 45].”
Annotation-efficient deep learning for automatic medical image segmentation (2020, background): “Methods such as data augmentation 41,46 , transfer learning 42,43 , semi-supervised learning 44,47 , and self-supervised learning 48,49 have been extensively investigated to handle cases with limited training annotations or no target domain annotations.”
Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks (2019, methodology): “Zhang et al . have used a complex 3D Cycle-GAN with an additional shape-consistency loss to enable modality transfer between cardiac MRI and cardiac CT by incorporating a subset of labeled data in both modalities 8 .”
Shifting machine learning for healthcare from development to deployment and from models to data (2022, methodology): “Limitations with geometric distortions can be overcome by using two auxiliary mappings to constrain the geometric invariance of synthetic data 21 .”
3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions (2019, background): “GANs for 3D Point Clouds Generation: GANs [13] for 2D image generation tasks have been widely studied with great success [10, 18, 23, 26, 33, 36, 42, 43, 49, 50], but GANs for 3D point cloud generation have rarely been studied in the computer vision field.”
Synthetic Data for Deep Learning (2019, background): “[704] learn a CycleGAN-based architecture [714] to learn volume-to-volume (i.”
Self-derived organ attention for unpaired CT-MRI deep domain adaptation based MRI segmentation (2020, methodology): “Improved I2I translations and segmentations have also been achieved by preserving the structure of interest’s (SOI) geometry by combining segmentation losses (Zhang et al 2018) or by combining high-level segmentation relevant feature matching losses (Jiang et al 2018a).”
SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining (2021, neutral): “Therefore, there is a clear need for a fast, accurate, and reproducible automated method, for segmentation of brain scans of any contrast and resolution, and that can adapt to a wide range of populations.”
Self-Supervised CycleGAN for Object-Preserving Image-to-Image Domain Adaptation (2020, methodology): “To address this problem, existing studies [34,10] proposed to use a segmentation sub-task with pixel-wise annotation as an auxiliary regularization to assist the training of the generators, which enabled CycleGAN to be applied to tasks such as domain adaptation [34] and data augmentation [10].”
Recent advances and clinical applications of deep learning in medical image analysis (2021, background): “For medical image segmentation, to alleviate the need for a large amount of annotated training data, reserachers have adopted generative models for image synthesis to increase the number of training examples (Zhang et al., 2018b; Zhao et al., 2019a).”
Sphere Generative Adversarial Network Based on Geometric Moment Matching (2019, methodology): “[8], generative adversarial networks (GANs) have attracted much research interest, and they have been used to achieve outstanding performance in a wide range of computer vision applications including in image generation [17, 26], super resolution [14], video prediction [19], style transfer [5, 12, 3”
Unsupervised Bidirectional Cross-Modality Adaptation via Deeply Synergistic Image and Feature Alignment for Medical Image Segmentation (2020, methodology): “Compared to [37] and [38], a key characteristic of our ap-”
A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises (2020, background): “[58] propose to synthesize and segment multimodal medical volumes using GANs with the cycle and shape consistencies.”