Disentangled representation learning in cardiac image analysis

By Agisilaos Chartsias, Thomas Joyce, G. Papanastasiou, S. Semple, M. Williams, D. Newby, Rohan Dharmakumar, Sotirios A. Tsaftaris. 2019

168 citations recorded by the source corpus.

Typically, a medical image offers spatial information on the anatomy (and pathology) modulated by imaging specific characteristics. Many imaging modalities including Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) can be interpreted in this way. We can venture further and consider that a medical image naturally factors into some spatial factors depicting anatomy and factors that denote the imaging characteristics. Here, we explicitly learn this decomposed (disentangled) representation of imaging data, focusing in particular on cardiac images. We propose Spatial Decomposition Network (SDNet), which factorises 2D medical images into spatial anatomical factors and non-spatial modality factors. We demonstrate that this high-level representation is ideally suited for several medical image analysis tasks, such as semi-supervised segmentation, multi-task segmentation and regression, and image-to-image synthesis. Specifically, we show that our model can match the performance of fully supervised segmentation models, using only a fraction of the labelled images. Critically, we show that our factorised representation also benefits from supervision obtained either when we use auxiliary tasks to train the model in a multi-task setting (e.g. regressing to known cardiac indices), or when aggregating multimodal data from different sources (e.g. pooling together MRI and CT data). To explore the properties of the learned factorisation, we perform latent-space arithmetic and show that we can synthesise CT from MR and vice versa, by swapping the modality factors. We also demonstrate that the factor holding image specific information can be used to predict the input modality with high accuracy. Code will be made available at https://github.com/agis85/anatomy_modality_decomposition.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Weakly-Supervised Disentanglement Without Compromises (2020, background): “This result is of great importance as the community is becoming increasingly interested in the practical benefits of disentangled representations (van Steenkiste et al., 2019; Locatello et al., 2019a; Creager et al., 2019; Chao et al., 2019; Iten et al., 2020; Chartsias et al., 2019; Higgins et al.,”
Multi-Centre, Multi-Vendor and Multi-Disease Cardiac Segmentation: The M&Ms Challenge (2021, background): “Lastly, one team (P12) proposed a modified Spatial Decomposition Network (SDN) [36] with an AdaIN [37] decoder.”
A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises (2020, methodology): “In medical imaging, unsupervised learning and disentanglement have been used in image registration [70], motion tracking [71], artifact reduction [72], improving classification [73], domain adaptation [74], and general modeling [75].”