Deep learning with multimodal representation for pancancer prognosis prediction

By Anika Cheerla, Olivier Gevaert. bioRxiv, 2019

363 citations recorded by the source corpus.

Estimating the future course of cancer is invaluable to physicians; however, current clinical methods fail to effectively use the vast amount of multimodal data that is available for cancer patients. To tackle this problem, we constructed a deep neural network based model to predict the survival of patients for 20 different cancer types using gene expressions, microRNA data, clinical data and histopathology whole slide images (WSIs). We developed an unsupervised encoder to compress these four data modalities into a single feature vector for each patient, handling missing data through a resilient, multimodal dropout method. Encoding methods were tailored to each data type - using deep highway networks to extract features from genomic and clinical data, and convolutional neural networks extract features from pathology images. We then used these feature encodings trained on pancancer data to predict pancancer and single cancer survival data, achieving a C-index of 0.784 overall. This work shows that it is possible to build a pancancer model for prognosis that also predicts prognosis in single cancer sites. Furthermore, our model handles multiple data modalities, efficiently analyzes WSIs, and summarizes patient details flexibly into an unsupervised, informative profile. We thus present a powerful automated tool to accurately determine prognosis, a key step towards personalized treatment for cancer patients.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/adffa8208c217dd9e293d669c5120cfe0c210ef1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis (2019, background): “Cheerla et al. [25] developed an unsupervised multimodal encoder network for integrating histology image and genomic modalities via concatenation that is resilient to missing data.”
Artificial intelligence for digital and computational pathology (2023, methodology): “To overcome this limitation, one option is to perform 3D reconstruction from serially sectioned and aligned H&E tissue sections to visualize the entire tissue at subcellular resolution, without the need for special sample preparation protocols”
Multimodal deep learning for biomedical data fusion: a review (2022, methodology): “Different approaches have been proposed, such as multi-task learning [70], generative models [63] and multimodal dropout [78, 101].”
Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond (2021, methodology): “In general, deep learning based AI algorithms have been developed for image-based classification [43], diagnosis [44–46] and prognosis [47,48], genome interpretation [49], biomarker discovery [50,51], monitoring by wearable life-logging devices [52], and automated robotic surgery [53] to enhance…”
Artificial Intelligence in Cancer Research and Precision Medicine (2021, positive): “For example, training predictive models on multiple integrated rather than singular data sources has been shown, for instance by Cheerla and Gevaert, to improve prediction of overall survival in patients across cancers (5).”
A foundation model for clinical-grade computational pathology and rare cancers detection (2024, positive): “Indeed, it has been shown that training a model on multiple tissues or cancer variants can improve detection performance for each cancer 53 , as many morphological features are observed across cancers from different topographies 54 .”
Harnessing multimodal data integration to advance precision oncology (2021, methodology): “The model achieves a concordance index (c-index) of 0.78 to stratify patients by OS 80 and is robust to missingness, but it conceptually encourages mutual information, potentially at the expense of complementary information gained via fusion methods (Fig.”