Multimodal MR Synthesis via Modality-Invariant Latent Representation

By Agisilaos Chartsias, Thomas Joyce, Mario Valerio Giuffrida, Sotirios A. Tsaftaris. IEEE Transactions on Medical Imaging, 2017

347 citations recorded by the source corpus.

We propose a multi-input multi-output fully convolutional neural network model for MRI synthesis. The model is robust to missing data, as it benefits from, but does not require, additional input modalities. The model is trained end-to-end, and learns to embed all input modalities into a shared modality-invariant latent space. These latent representations are then combined into a single fused representation, which is transformed into the target output modality with a learnt decoder. We avoid the need for curriculum learning by exploiting the fact that the various input modalities are highly correlated. We also show that by incorporating information from segmentation masks the model can both decrease its error and generate data with synthetic lesions. We evaluate our model on the ISLES and BRATS data sets and demonstrate statistically significant improvements over state-of-the-art methods for single input tasks. This improvement increases further when multiple input modalities are used, demonstrating the benefits of learning a common latent space, again resulting in a statistically significant improvement over the current best method. Finally, we demonstrate our approach on non skull-stripped brain images, producing a statistically significant improvement over the previous best method. Code is made publicly available at https://github.com/agis85/multimodal_brain_synthesis.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/63881dd34e35354aafc2671568b959298a0cac37

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Unsupervised Medical Image Translation With Adversarial Diffusion Models (2022, background): “Learning-based image translation involves network models trained to capture a prior on the conditional distribution of target given source images [18], [19], [20].”
Image Synthesis in Multi-Contrast MRI With Conditional Generative Adversarial Networks (2018, methodology): “Overall, both proposed methods yield synthetic images of remarkable visual similarity to reference images, and pGAN visually and quantitatively improves synthesis quality compared to stateof-the-art methods [21], [23].”
Small Sample Learning in Big Data Era (2018, background): “Afterwards, Chartsias et al. (2018) extended Joyce et al. (2017)’s work, easily predicting new output modalities through the addition of decoders which can be trained in isolation.”
Hi-Net: Hybrid-Fusion Network for Multi-Modal MR Image Synthesis (2020, methodology): “In our comparison, we use the paired data to synthesize medical images from one modality to another; and 3) MM-Syns [48].”
ResViT: Residual Vision Transformers for Multimodal Medical Image Synthesis (2021, positive): “Earlier studies using CNNs for this purpose reported significant improvements over traditional approaches [22], [23], [25]–[28].”
SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining (2021, neutral): “Then, the background region is split into a random number of N regions ( N ∈ [3 , 10]), which aim at representing the surrounding structures with di ﬀ erent levels of granularity (Supplement 6).”
Diffusion models in medical imaging: A comprehensive survey (2022, methodology): “Experiments on the BraTS19 dataset [80, 81, 82], which contains four MRI modalities for each subject, show that the UMM-CSGM is capable of generating missing-modality images with higher fidelity and structural information of the brain tissue compared to SOTA methods [83, 84, 85, 86, 87].”
Deep learning in radiology: an overview of the concepts and a survey of the state of the art with focus on MRI (2018, methodology): “In another approach [114] in order to synthesize brain MRI images based on other MRI sequences in the same patient, convolutional encoders were built to generate a latent representation of images.”