3D Deep Learning for Multi-modal Imaging-Guided Survival Time Prediction of Brain Tumor Patients

By Dong Nie, Han Zhang, Ehsan Adeli, Luyan Liu, Dinggang Shen. 2016

269 citations recorded by the source corpus.

High-grade glioma is the most aggressive and severe brain tumor that leads to death of almost 50% patients in 1–2 years. Thus, accurate prognosis for glioma patients would provide essential guidelines for their treatment planning. Conventional survival prediction generally utilizes clinical information and limited handcrafted features from magnetic resonance images (MRI), which is often time consuming, laborious and subjective. In this paper, we propose using deep learning frameworks to automatically extract features from multi-modal preoperative brain images (i.e., T1 MRI, fMRI and DTI) of high-grade glioma patients. Specifically, we adopt 3D convolutional neural networks (CNNs) and also propose a new network architecture for using multi-channel data and learning supervised features. Along with the pivotal clinical features, we finally train a support vector machine to predict if the patient has a long or short overall survival (OS) time. Experimental results demonstrate that our methods can achieve an accuracy as high as 89.9% We also find that the learned features from fMRI and DTI play more important roles in accurately predicting the OS time, which provides valuable insights into functional neuro-oncological applications.

Subfields: Computer Vision and Pattern Recognition, Genetics, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Genetics

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Opportunities and obstacles for deep learning in biology and medicine (2017, background): “[63] showed that multimodal, multi-channel 3D deep architecture was successful at learning high-level brain tumor appearance features jointly from MRI, functional MRI, and diffusion MRI images, outperforming single-modality or 2D models.”
A survey on deep learning in medical image analysis (2017, background): “In contrast, Nie et al. (2016c) exploited the 3D nature of MRI by training a 3D CNN to assess survival in patients su ﬀ ering from high-grade gliomas.”
Deep Learning for Health Informatics (2017, methodology): “Deep learning provides the possibility to automate and merge the extraction of relevant features with the classification procedure [55], [65].”
Hi-Net: Hybrid-Fusion Network for Multi-Modal MR Image Synthesis (2020, background): “[44] proposed a 3D deep learning model to predict the overall survival time for brain gliomas patients by fusing T1 MRI, functional (fMRI) and diffusion tensor imaging (DTI) data.”
Applications of Deep Learning and Reinforcement Learning to Biological Data (2017, methodology): “CNN was also used extensively: on CT scans to detect– anatomical structure [136], sclerotic metastases of spine along with colonic polyps and lymph nodes (LN) [137], thoracoabdominal LN and interstitial lung disease (ILD) [139], pulmonary nodules [138,140,141]; on (f)MRI and diffusion tensor images ”
Interpretable LSTMs For Whole-Brain Neuroimaging Analyses (2018, background): “, [22, 10, 23, 24, 25, 18, 26]), three major obstacles have so far prevented broad DL usage: (1) No structured framework for the application of DL models to whole-brain neuroimaging data exists.”
Deep Learning for Spatio-Temporal Data Mining: A Survey (2019, methodology): “Application domains Related works Transportation [177], [187], [116], [31], [29], [6], [32], [127], [53], [112], [26], [105], [166], [82], [23], [66], [108], [126], [81], [139], [78], [128], [146], [148], [72], [98], [189], [22], [24], [99], [96], [16], [97], [85], [15], [164], [165], [95], [140], [”
Current Applications and Future Impact of Machine Learning in Radiology. (2018, unknown): “For instance, brain tumor response to therapy can be estimated accurately with machine learning (92).”
Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images (2022, background): “In addition to surgical applications, characterization of delineated tumors can be directly utilized for the prediction of life expectancy [33].”
Analyzing Neuroimaging Data Through Recurrent Deep Learning Models (2018, unknown): “While researchers have started exploring the application of DL models to neuroimaging data (e.g., Plis et al., 2014; Suk et al., 2014; Nie et al., 2016; Sarraf and Tofighi, 2016; Mensch et al., 2018; Petrov et al., 2018; Yousefnezhad and Zhang, 2018), two major challenges have so far prevented broad”
3D Deep Learning on Medical Images: A Review (2020, background): “[70] take advantage of the 3D aspect of MRI through training a 3D CNN to evaluate the survival in patients going”