Manipulating Medical Image Translation with Manifold Disentanglement

By Siyu Liu, Jason A. Dowling, C. Engstrom, P. Greer, Stuart Crozier, Shekhar S. Chandra. 2020

8 citations recorded by the source corpus.

Most current medical image translation Generative Adversarial Networks (GANs) can only generate one definitive output for each input. However, in reality, there can be many simultaneously valid translations. For example, when synthesising medical images from segmentation, diverse tissue structures, contrasts, textures, and even modalities are possible based on the same segmentation. In this work, we propose Manifold Disentanglement Generative Adversarial Network (MDGAN), a style-based network capable of capturing this output diversity. The mechanism enabling output diversity is a style-based manifold, which is learnt from image data, and can be sampled to “stylise” the input into diverse outputs. We train MDGAN for segmentation-to-MR-and-CT translation and show that the manifold i) can learn distinct clusters that control the output modality (CT or MR), ii) can be traversed to smoothly alter features within each modality (such as tissue structures, contrasts in MR) and iii) is disentangled such that the input’s anatomical structures are faithfully preserved when generating diverse images based on the same segmentation map.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yujiu Yang
2. Ming-Hsuan Yang
3. Weihao Xia
4. Yulun Zhang
5. Jing-Hao Xue
6. Luc Van Gool
7. Bolei Zhou
8. Lei Zhang

In Media Technology

1. Yulun Zhang
2. Ming-Hsuan Yang
3. Yujiu Yang
4. Weihao Xia
5. Bolei Zhou
6. Jing-Hao Xue
7. Radu Timofte
8. Luc Van Gool

In Generative Models

1. Yujiu Yang
2. Weihao Xia
3. Jing-Hao Xue
4. Yulun Zhang
5. Ming-Hsuan Yang
6. Bolei Zhou
7. Daniel Cohen-Or
8. Dani Lischinski

Citations in the corpus

• GAN Inversion: A Survey (2021, background): “In addition to common applications in image processing, in the last few months, GAN inversion techniques have been widely introduced to many other tasks, such as 3D reconstruction [139], [140], image understanding [141], [142], multimodal learning [133], [134], [135], [143], [144], and medical imagi”