Predicting cancer outcomes from histology and genomics using convolutional networks

By Pooya Mobadersany, Safoora Yousefi, M. Amgad, D. Gutman, Jill S. Barnholtz-Sloan, José E. Velázquez Vega, D. Brat, Lee A. D. Cooper. Proceedings of the National Academy of Sciences of the United States of America, 2017

948 citations recorded by the source corpus.

Significance Predicting the expected outcome of patients diagnosed with cancer is a critical step in treatment. Advances in genomic and imaging technologies provide physicians with vast amounts of data, yet prognostication remains largely subjective, leading to suboptimal clinical management. We developed a computational approach based on deep learning to predict the overall survival of patients diagnosed with brain tumors from microscopic images of tissue biopsies and genomic biomarkers. This method uses adaptive feedback to simultaneously learn the visual patterns and molecular biomarkers associated with patient outcomes. Our approach surpasses the prognostic accuracy of human experts using the current clinical standard for classifying brain tumors and presents an innovative approach for objective, accurate, and integrated prediction of patient outcomes. Cancer histology reflects underlying molecular processes and disease progression and contains rich phenotypic information that is predictive of patient outcomes. In this study, we show a computational approach for learning patient outcomes from digital pathology images using deep learning to combine the power of adaptive machine learning algorithms with traditional survival models. We illustrate how these survival convolutional neural networks (SCNNs) can integrate information from both histology images and genomic biomarkers into a single unified framework to predict time-to-event outcomes and show prediction accuracy that surpasses the current clinical paradigm for predicting the overall survival of patients diagnosed with glioma. We use statistical sampling techniques to address challenges in learning survival from histology images, including tumor heterogeneity and the need for large training cohorts. We also provide insights into the prediction mechanisms of SCNNs, using heat map visualization to show that SCNNs recognize important structures, like microvascular proliferation, that are related to prognosis and that are used by pathologists in grading. These results highlight the emerging role of deep learning in precision medicine and suggest an expanding utility for computational analysis of histology in the future practice of pathology.

Subfields: Artificial Intelligence, Biophysics, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/7b05161938f4e5f89ddd7ab7f432de8ecab1566e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Multimodal deep learning for biomedical data fusion: a review (2022, background): “[77] showed that imbalance does not necessarily lead to poor performance if the input features of the lower dimensional modality are chosen with prior knowledge.”
Artificial intelligence for digital and computational pathology (2023, background): “For example, AI systems have been developed to predict response to immunotherapies 164–167 , targeted therapy 168,169 and chemotherapy 170 .”
Artificial intelligence as the next step towards precision pathology (2020, background): “Model interpretation might also reveal new hallmarks of a disease, such as the histologic presence of oedema in gliomas that has not been previously recognized as an unfavourable marker, but was detected by AI [101].”
Emerging role of deep learning‐based artificial intelligence in tumor pathology (2020, background): “It was demonstrated that DL approaches could predict the prognostic risk by learning histologic features in lung adenocarcinoma [76] and glioma [77].”
Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images (2021, methodology): “[42] uses vector concatenation to integrate histology and genomic features for survival outcome prediction.”
AI in Medical Imaging Informatics: Current Challenges and Future Directions (2020, positive): “These datasets can be correlated to molecular tumor characteristics and can be used to quantitatively characterize tissue at multiple spatial scales to create biomarkers that predict outcome and treatment response [150], [152]–[154].”
Deep learning with noisy labels: exploring techniques and remedies in medical image analysis (2019, background): “level image processing tasks such as denoising, enhancement, and reconstruction [6], to more high-level image analysis tasks such as segmentation, detection, classification, and registration [7], [8], and even more challenging tasks such as discovering links between the content of medical images and”
Harnessing multimodal data integration to advance precision oncology (2021, methodology): “Three-dimensional CNNs have shown success in stratifying patients with non- small-cell lung cancer (NSCLC) by overall survival (OS) 26 , and empirically outperformed two-dimensional CNNs in other radiology tasks, such as diagnosing appendicitis 27 .”
Artificial intelligence in clinical and genomic diagnostics (2019, background): “More generally, AI-based computer vision systems appear to be capable of predicting the genomic aberrations that are likely to be present in an individual ’ s genome on the basis of the complex phenotypes embedded in relevant clinical images [20, 75].”
Machine Learning for Precision Medicine. (2020, neutral): “(Mobadersany et al., 2018) Also: high-content screening, digital pathology, radiomics, skin diagnosis, EHR, .”