Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis

By Richard J. Chen, Ming Y. Lu, Jingwen Wang, Drew F. K. Williamson, S. Rodig, N. Lindeman, Faisal Mahmood. IEEE Transactions on Medical Imaging, 2019

654 citations recorded by the source corpus.

Cancer diagnosis, prognosis, mymargin and therapeutic response predictions are based on morphological information from histology slides and molecular profiles from genomic data. However, most deep learning-based objective outcome prediction and grading paradigms are based on histology or genomics alone and do not make use of the complementary information in an intuitive manner. In this work, we propose Pathomic Fusion, an interpretable strategy for end-to-end multimodal fusion of histology image and genomic (mutations, CNV, RNA-Seq) features for survival outcome prediction. Our approach models pairwise feature interactions across modalities by taking the Kronecker product of unimodal feature representations, and controls the expressiveness of each representation via a gating-based attention mechanism. Following supervised learning, we are able to interpret and saliently localize features across each modality, and understand how feature importance shifts when conditioning on multimodal input. We validate our approach using glioma and clear cell renal cell carcinoma datasets from the Cancer Genome Atlas (TCGA), which contains paired whole-slide image, genotype, and transcriptome data with ground truth survival and histologic grade labels. In a 15-fold cross-validation, our results demonstrate that the proposed multimodal fusion paradigm improves prognostic determinations from ground truth grading and molecular subtyping, as well as unimodal deep networks trained on histology and genomic data alone. The proposed method establishes insight and theory on how to train deep networks on multimodal biomedical data in an intuitive manner, which will be useful for other problems in medicine that seek to combine heterogeneous data streams for understanding diseases and predicting response and resistance to treatment. Code and trained models are made available at: https://github.com/mahmoodlab/PathomicFusion.

Subfields: Artificial Intelligence, Biophysics, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/96f8e0fd55f456efe502c350ecc543522c795454

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Graph Representation Learning in Biomedicine and Healthcare (2021, positive): “Information on cell morphology and tissue microarchitecture can be aggregated into cell graphs to, for instance, grade cancer histology images (for example, using GNNs) 8,176–178 .”
Multimodal deep learning for biomedical data fusion: a review (2022, methodology): “[79] enabled modality-specific interpretability by applying Grad-CAM [97] for WSI and integrated gradient [98] for cell graph and genomic modalities using convolutional, graph convolutional [14] and fully connected branches.”
Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images (2021, methodology): “As multimodal comparisons to MCAT, we augment the previous set-based network architectures with two common late fusion mechanisms to integrate bag-level WSI features and genomic features: 1) concatenation [42], and 2) bilinear pooling [19, 65, 57, 9].”
Multimodal data fusion for cancer biomarker discovery with deep learning (2023, background): “Similar efforts as for multi-omics data fusion have been explored for multiscale data 89,102–107 .”
Artificial intelligence for digital and computational pathology (2023, methodology): “Deep learning in pathology has also been used for developing AI-based assistive tools that can extract actionable objects and representations”
Harnessing multimodal data integration to advance precision oncology (2021, methodology): “…histopathology include cellular morphological features and mRNA-seq data integration in NSCLC 82 , combined histological and gene expression features in breast cancer 83 , and histopathological and genomic features in glioma using genomic survival CNNs 84 and tensor fusion networks (TFNs) 85 .”
The impact of site-specific digital histology signatures on deep learning model accuracy and bias (2021, background): “Studies demonstrating histologic discrimination of survival and recurrence in glioblastoma 52,54 , renal cell cancer 52 , and lung cancer 55 patients from TCGA which lack external validation cohorts may therefore have biased estimates of the outcome.”