Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis

By Heung-Il Suk, Seong-Whan Lee, Dinggang Shen. 2014

913 citations recorded by the source corpus.

For the last decade, it has been shown that neuroimaging can be a potential tool for the diagnosis of Alzheimer’s Disease (AD) and its prodromal stage, Mild Cognitive Impairment (MCI), and also fusion of different modalities can further provide the complementary information to enhance diagnostic accuracy. Here, we focus on the problems of both feature representation and fusion of multimodal information from Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET). To our best knowledge, the previous methods in the literature mostly used hand-crafted features such as cortical thickness, gray matter densities from MRI, or voxel intensities from PET, and then combined these multimodal features by simply concatenating into a long vector or transforming into a higher-dimensional kernel space. In this paper, we propose a novel method for a high-level latent and shared feature representation from neuroimaging modalities via deep learning. Specifically, we use Deep Boltzmann Machine (DBM)1, a deep network with a restricted Boltzmann machine as a building block, to find a latent hierarchical feature representation from a 3D patch, and then devise a systematic method for a joint feature representation from the paired patches of MRI and PET with a multimodal DBM. To validate the effectiveness of the proposed method, we performed experiments on ADNI dataset and compared with the state-of-the-art methods. In three binary classification problems of AD vs. healthy Normal Control (NC), MCI vs. NC, and MCI converter vs. MCI non-converter, we obtained the maximal accuracies of 95.35%, 85.67%, and 74.58%, respectively, outperforming the competing methods. By visual inspection of the trained model, we observed that the proposed method could hierarchically discover the complex latent patterns inherent in both MRI and PET.

Subfields: Computer Vision and Pattern Recognition, Neurology, Psychiatry and Mental health

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Neurology

In Psychiatry and Mental health

Citations in the corpus

Deep Learning for Health Informatics (2017, background): “[59] proposed a DBM to extract a latent hierarchical feature representation from 3-D patches of”
State-space model with deep learning for functional dynamics estimation in resting-state fMRI (2016, neutral): “…along with recent successes in various application domains including neuroimaging analysis (Hjelm et al., 2014; Plis et al., 2014; Wu et al., 2015; Suk et al., 2014, 2015b; Kim et al., 2016), we devise a deep architecture by stacking multiple RBMs to transform the input features of an fMRI at a…”
Representation learning for mammography mass lesion classification with convolutional neural networks (2016, neutral): “[14] H.”
Residual and plain convolutional neural networks for 3D brain MRI classification (2017, methodology): “[5] used a complex architecture of deep belief network (DBN) with patch-sampling preprocessing and a combination of MRI and PET modalities with a weighted ensemble of SVM classifiers on top of the network.”
Multimodal deep learning for biomedical data fusion: a review (2022, methodology): “[55] applied a multimodal deep boltzmann machine (DBM) [95] to predict AD from magnetic resonance images (MRI) and positron emission tomography (PET) scans.”
Applications of Deep Learning and Reinforcement Learning to Biological Data (2017, methodology): “Deep Boltzmann Machine DBM (Fig.”
Deep Learning-based Image Segmentation on Multimodal Medical Imaging (2019, background): “The multimodal feature representation framework introduced in [26] fuses information from MRI and PET in a hierarchical deep learning approach.”
Multimodal Neuroimaging Feature Learning With Multimodal Stacked Deep Polynomial Networks for Diagnosis of Alzheimer's Disease (2018, methodology): “M-RBM [20] 93 AD, 76 MCI-C, 128 MCI-NC, 101 NC MRI + PET The pioneering multimodal deep RBM (M-RBM) based feature learning algorithms”
A deep learning framework for supporting the classification of breast lesions in ultrasound images (2017, background): “diagnosis for Alzheimers disease[46, 47, 48, 49] and mild cognitive impairment[47], 78”
Multimodal and Multiscale Deep Neural Networks for the Early Diagnosis of Alzheimer’s Disease using structural MR and FDG-PET images (2017, methodology): “…into cortical and subcortical gray-matter compartments, and further subdividing each into patches of a hierarchical size, and extract features from each-sized patch 26–28 by averaging within the patch and use these multi-scale features taken from multiple modalities into a deep learning framework.”