Medical Image Synthesis with Context-Aware Generative Adversarial Networks

By Dong Nie, Roger Trullo, Caroline Petitjean, Su Ruan, Dinggang Shen. 2016

763 citations recorded by the source corpus.

Computed tomography (CT) is critical for various clinical applications, e.g., radiation treatment planning and also PET attenuation correction in MRI/PET scanner. However, CT exposes radiation during acquisition, which may cause side effects to patients. Compared to CT, magnetic resonance imaging (MRI) is much safer and does not involve radiations. Therefore, recently researchers are greatly motivated to estimate CT image from its corresponding MR image of the same subject for the case of radiation planning. In this paper, we propose a data-driven approach to address this challenging problem. Specifically, we train a fully convolutional network (FCN) to generate CT given the MR image. To better model the nonlinear mapping from MRI to CT and produce more realistic images, we propose to use the adversarial training strategy to train the FCN. Moreover, we propose an image-gradient-difference based loss function to alleviate the blurriness of the generated CT. We further apply Auto-Context Model (ACM) to implement a context-aware generative adversarial network. Experimental results show that our method is accurate and robust for predicting CT images from MR images, and also outperforms three state-of-the-art methods under comparison.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Visual Feature Attribution Using Wasserstein GANs (2017, methodology): “Although some of the above models use 3D data, the examined volumes are usually relatively small [23], or the networks operate in a patch-wise fashion [40].”
LOGAN: Membership Inference Attacks Against Generative Models (2017, background): “(Note that image synthesis is commonly used to create datasets for healthcare applications [44, 13].”
LOGAN: Evaluating Privacy Leakage of Generative Models Using Generative Adversarial Networks (2017, methodology): “Note that image synthesis is already a common technique for creating large datasets in the field of healthcare [29, 8, 9].”
Synthetic data augmentation using GAN for improved liver lesion classification (2018, background): “Recently, several medical imaging applications have applied the GAN framework [6, 7, 8].”
Generative Adversarial Network in Medical Imaging: A Review (2018, neutral): “3T→ 7T MR Nie et al. (2018) Cascade GAN L1, 2, 4 – M9, 10 [Ë] Brain”
Medical Image Synthesis with Deep Convolutional Adversarial Networks (2018, background): “A preliminary version of this work has been presented at a conference [37].”
CovidGAN: Data Augmentation Using Auxiliary Classifier GAN for Improved Covid-19 Detection (2020, methodology): “A patch-based GAN was developed in Nie et al. [14] to convert brain CT pictures to the corresponding MRI.”
GAN-based synthetic brain MR image generation (2018, background): “Therefore, many medical imaging researchers have begun to use GANs recently, such as in image super-resolution [16], anomaly detection [17], and estimating CT images from the corresponding MR images [18].”
Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networks (2018, methodology): “GANs have been used to generate a motion model from a single preoperative MRI [10], upsample a lowresolution fundus image [13], create a synthetic head CT from a brain MRI [16], and synthesizing T2-weight MRI from T1-weighted ones (and vice-versa) [5].”
Tumor-Aware, Adversarial Domain Adaptation from CT to MRI for Lung Cancer Segmentation (2018, methodology): “Cross-domain adaptation of highly different modalities, has been applied in medical image analysis for image synthesis using paired images [6] and unpaired images [7], as well as for segmentation [8, 9].”
Deep learning in medical imaging and radiation therapy. (2018, background): “259 Han et al. 256 adopted and modified the U-net architecture for sCT generation from MRI.”
Learning Deep Representations of Medical Images using Siamese CNNs with Application to Content-Based Image Retrieval (2017, background): “…as classiﬁcation [Esteva et al., 2017, Gulshan et al., 2016], segmentation [Havaei et al., 2017, Guo et al., 2017], image generation and synthesis [Nie et al., 2017, Van Nguyen et al., 2015], image captioning [Moradi et al., 2016, Shin et al., 2015], and CBMIR [Sun et al., 2017, Anavi et al.,…”
Medical Image Synthesis with Generative Adversarial Networks for Tissue Recognition (2018, background): “, polling layers are not involved in the developed GAN model [17].”
Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network (2018, methodology): “To synthesize medical images, recent advances [24, 6] have used generative adversarial networks (GANs) [10] to formulate it as an image-to-image translation task.”
Deep Learning Applications in Medical Image Analysis (2018, methodology): “[106], [107], in which GANs were used to generate CT brain images from MRI images.”
Synthetic Data for Deep Learning (2019, methodology): “Similar image-to-image translation techniques have been applied to generating images from 2D MR brain images to CT and back [282, 424, 639], PET to CT [27], cardiac CT to MR [96], virtual”