HeMIS: Hetero-Modal Image Segmentation

By Mohammad Havaei, Nicolas Guizard, Nicolas Chapados, Yoshua Bengio. International Conference on Medical Image Computing and Computer-Assisted Intervention, 2016

355 citations recorded by the source corpus.

We introduce a deep learning image segmentation framework that is extremely robust to missing imaging modalities. Instead of attempting to impute or synthesize missing data, the proposed approach learns, for each modality, an embedding of the input image into a single latent vector space for which arithmetic operations (such as taking the mean) are well defined. Points in that space, which are averaged over modalities available at inference time, can then be further processed to yield the desired segmentation. As such, any combinatorial subset of available modalities can be provided as input, without having to learn a combinatorial number of imputation models. Evaluated on two neurological MRI datasets (brain tumors and MS lesions), the approach yields state-of-the-art segmentation results when provided with all modalities; moreover, its performance degrades remarkably gracefully when modalities are removed, significantly more so than alternative mean-filling or other synthesis approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neurology

View Paper: https://www.semanticscholar.org/paper/1a07daa88a366c10afa80cbdf63ad1b5d0759d1e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neurology

Citations in the corpus

Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge (2018, background): “Alternative research directions include the enhancement of the flexibility of DL systems that might lack a given set of input images [139], as a transition measure towards worldwide adoption of the standardization initiatives for GBM imaging [140].”
ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases (2017, background): “Recent notable work includes (but do not limit to) an overview review on the future promise of deep learning [14] and a collection of important medical applications on lymph node and interstitial lung disease detection and classification [36, 42]; cerebral microbleed detection [11]; pulmonary nodule”
SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining (2021, neutral): “This problem can partly be ameliorated by training on multi-modality scans with modality dropout [30], which results in a network able to individually segment each training modality, but that still cannot be applied to unseen domains.”
HyperDense-Net: A Hyper-Densely Connected CNN for Multi-Modal Image Segmentation (2018, background): “They achieved state-of-the-art performances in a broad range of medical image segmentation problems [36], [37], including multimodal tasks [14]–[16], [25]–[27], [29], [32], [33], [38], [39].”
Improving automated multiple sclerosis lesion segmentation with a cascaded 3D convolutional neural network approach (2017, background): “Furthermore, Havaei et al. (2016) have also introduced another lesion segmentation framework with independent image modality convolution pipelines that reduces the eﬀect of missing modalities of new unseen examples.”
A review on deep learning in medical image analysis (2021, background): “The k-nearest neighbours algorithm is a classic example, where the transductive algorithm uses it directly each time a prediction is required, instead of modelling the training data [3, 47, 72].”
Automatic Brain Tumor Segmentation Using Cascaded Anisotropic Convolutional Neural Networks (2017, methodology): “These include efficient end-to-end training using a fully convolutional approach [1,8], incorporating large visual contexts by employing a mixture of convolution and downsampling operations [12,9], maintaining high resolution multi-scale features with dilated convolution and residual connection [17,”
Multimodal MR Synthesis via Modality-Invariant Latent Representation (2017, methodology): “This method was shown in [26] to work very well for image segmentation, producing state of the art results.”
Longitudinal multiple sclerosis lesion segmentation: Resource and challenge (2017, neutral): “…et al., 2015, 2016; Deshpande et al., 2015; Dugas-Phocion et al., 2004; Elliott et al., 2013, 2014; Ferrari et al., 2003; Geremia et al., 2010; Havaei et al., 2016; Jain et al., 2015; Jog et al., 2015; Johnston et al., 1996; Kamber et al., 1996; Khayati et al., 2008; Rey et al., 1999, 2002;…”