Hi-Net: Hybrid-Fusion Network for Multi-Modal MR Image Synthesis

By Tao Zhou, Huan Fu, Geng Chen, Jianbing Shen, Ling Shao. IEEE Transactions on Medical Imaging, 2020

363 citations recorded by the source corpus.

Magnetic resonance imaging (MRI) is a widely used neuroimaging technique that can provide images of different contrasts (i.e., modalities). Fusing this multi-modal data has proven particularly effective for boosting model performance in many tasks. However, due to poor data quality and frequent patient dropout, collecting all modalities for every patient remains a challenge. Medical image synthesis has been proposed as an effective solution, where any missing modalities are synthesized from the existing ones. In this paper, we propose a novel Hybrid-fusion Network (Hi-Net) for multi-modal MR image synthesis, which learns a mapping from multi-modal source images (i.e., existing modalities) to target images (i.e., missing modalities). In our Hi-Net, a modality-specific network is utilized to learn representations for each individual modality, and a fusion network is employed to learn the common latent representation of multi-modal data. Then, a multi-modal synthesis network is designed to densely combine the latent representation with hierarchical features from each modality, acting as a generator to synthesize the target images. Moreover, a layer-wise multi-modal fusion strategy effectively exploits the correlations among multiple modalities, where a Mixed Fusion Block (MFB) is proposed to adaptively weight different fusion strategies. Extensive experiments demonstrate the proposed model outperforms other state-of-the-art medical image synthesis methods.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/e44da625e6af8551402f645709cdd7672d06b099

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Ling Shao
2. Deng-Ping Fan
3. Jianbing Shen
4. Ge-Peng Ji
5. Huan Fu
6. Tao Zhou
7. Ming-Ming Cheng
8. Wenguan Wang

In Media Technology

1. Ling Shao
2. Jianbing Shen
3. Tao Zhou
4. Dorit Merhof
5. Reza Azad
6. Ilker Hacihaliloglu
7. Huan Fu
8. Yi Zhou

Citations in the corpus

• ResViT: Residual Vision Transformers for Multimodal Medical Image Synthesis (2021, background): “Further improvements were attained by leveraging enhanced architectural designs [36]–[39], and learning strategies [40]–[42].”
• Diffusion models in medical imaging: A comprehensive survey (2022, methodology): “Experiments on the BraTS19 dataset [80, 81, 82], which contains four MRI modalities for each subject, show that the UMM-CSGM is capable of generating missing-modality images with higher fidelity and structural information of the brain tissue compared to SOTA methods [83, 84, 85, 86, 87].”
• Inf-Net: Automatic COVID-19 Lung Infection Segmentation From CT Images (2020, methodology): “performance. In future work, we will study to construct an end-to-end framework to achieve this task. Besides, due to the limited size of dataset, we will use the Generative Adversarial Network (GAN) [82] or Conditional Variational Autoencoders (CVAE) [83] to synthesize more samples, which can be re”