Pan-Cancer Integrative Histology-Genomic Analysis via Multimodal Deep Learning

By Richard J. Chen, Ming Y. Lu, Drew F. K. Williamson, Tiffany Y. Chen, Jana Lipková, Zahra Noor, M. Shaban, Maha Shady, Mane Williams, Bumjin Joo and 1 more. Cancer Cell, 2022

577 citations recorded by the source corpus.

Summary The rapidly emerging field of computational pathology has demonstrated promise in developing objective prognostic models from histology images. However, most prognostic models are either based on histology or genomics alone and do not address how these data sources can be integrated to develop joint image-omic prognostic models. Additionally, identifying explainable morphological and molecular descriptors from these models that govern such prognosis is of interest. We use multimodal deep learning to jointly examine pathology whole slide images and molecular profile data from 14 cancer types. Our weakly-supervised, multimodal deep learning algorithm is able to fuse these heterogeneous modalities to predict outcomes and discover prognostic features that correlate with poor and favorable outcomes. We present all analyses for morphological and molecular correlates of patient prognosis across the 14 cancer types at both a disease and patient-level in an interactive open-access database (http://pancancer.mahmoodlab.org) to allow for further exploration, biomarker discovery and feature assessment.

Subfields: Artificial Intelligence, Cancer Research, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/690745fe87697e8042b38e9d98fd263e4085ccae

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cancer Research

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Multimodal data fusion for cancer biomarker discovery with deep learning (2023, background): “However, multimodal approaches are starting to emerge with encouraging solutions not only for interpretability but also for discovery of associations between modalities 147,150 .”
Artificial intelligence for digital and computational pathology (2023, background): “Since its inception, CPath has pursued two primary and non-mutually exclusive goals: automating portions of the routine clinical work-flow and gaining new insights using data from that same workflow, sometimes supplemented by additional data sources (Fig.”
Algorithm fairness in artificial intelligence for medicine and healthcare (2023, methodology): “In cancer prognosis, although useful in highlighting important regions-of-interest in histology tissue, it is unclear how highly attributed pixel regions can be used by clinicians for patient stratification without further post-hoc assessment 322 .”