Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study

By Jakob Nikolas Kather, J. Krisam, Pornpimol Charoentong, Tom Luedde, E. Herpel, Cleo-Aron Weis, T. Gaiser, A. Marx, Nektarios A. Valous, D. Ferber and 8 more. PLoS Medicine, 2019

1,042 citations recorded by the source corpus.

Background For virtually every patient with colorectal cancer (CRC), hematoxylin–eosin (HE)–stained tissue slides are available. These images contain quantitative information, which is not routinely used to objectively extract prognostic biomarkers. In the present study, we investigated whether deep convolutional neural networks (CNNs) can extract prognosticators directly from these widely available images. Methods and findings We hand-delineated single-tissue regions in 86 CRC tissue slides, yielding more than 100,000 HE image patches, and used these to train a CNN by transfer learning, reaching a nine-class accuracy of >94% in an independent data set of 7,180 images from 25 CRC patients. With this tool, we performed automated tissue decomposition of representative multitissue HE images from 862 HE slides in 500 stage I–IV CRC patients in the The Cancer Genome Atlas (TCGA) cohort, a large international multicenter collection of CRC tissue. Based on the output neuron activations in the CNN, we calculated a “deep stroma score,” which was an independent prognostic factor for overall survival (OS) in a multivariable Cox proportional hazard model (hazard ratio [HR] with 95% confidence interval [CI]: 1.99 [1.27–3.12], p = 0.0028), while in the same cohort, manual quantification of stromal areas and a gene expression signature of cancer-associated fibroblasts (CAFs) were only prognostic in specific tumor stages. We validated these findings in an independent cohort of 409 stage I–IV CRC patients from the “Darmkrebs: Chancen der Verhütung durch Screening” (DACHS) study who were recruited between 2003 and 2007 in multiple institutions in Germany. Again, the score was an independent prognostic factor for OS (HR 1.63 [1.14–2.33], p = 0.008), CRC-specific OS (HR 2.29 [1.5–3.48], p = 0.0004), and relapse-free survival (RFS; HR 1.92 [1.34–2.76], p = 0.0004). A prospective validation is required before this biomarker can be implemented in clinical workflows. Conclusions In our retrospective study, we show that a CNN can assess the human tumor microenvironment and predict prognosis directly from histopathological images.

Subfields: Artificial Intelligence, Biophysics, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/a4eb7cc590b65e6be34fb74a4568d661385f28a2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, methodology): “PathMNIST is adapted from a dataset based on kather’s work [64].”
A deep learning model to predict RNA-Seq expression of tumours from whole slide images (2020, neutral): “…correlation above significance threshold (BH correction) d b c e –log(p-value) 0 2 4 6 8 10 80 20 40 60 80 100 60 WSI Transcriptomic representation Kather et al. 2019 Adapted from Kather et al. 2019 Adapted from Kather et al. 2019 40 20 0 FFPE slides: No. patients = 465 (No. slides = 483) No.”
Deep learning in histopathology: the path to the clinic (2021, methodology): “Other examples of biomarkers include those related to the amount of intratumoral stroma 71 , such as the tumor–stroma ratio 72 , which can be assessed by computing the ratio between the tumor and tumor-associated stroma obtained via image segmentation, and the quantification of programmed…”
Artificial intelligence in histopathology: enhancing cancer research and clinical oncology (2022, methodology): “A similar, more model-centric approach is through model attention heatmaps 29 or deep dream images 48 that can reveal informative image features.”
Clinical-grade Detection of Microsatellite Instability in Colorectal Tumors by Deep Learning. (2020, methodology): “periments, all image tiles were color normalized with the Macenko method(29) as described previously.(30) A modified”
MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification (2021, methodology): “The PathMNIST is based on a prior study 16,17 for predicting survival from colorectal cancer histology slides, providing a dataset”
Deep neural network models for computational histopathology: A survey (2019, background): “, 2019) and colorectal cancer (Kather et al., 2019; Bychkov et al., 2018), and risk of recurrence in breast cancer (Couture et al.”
The Application of Deep Learning in Cancer Prognosis Prediction (2020, methodology): “[74] hand labeled 100,000 image patches using 86 CRC H&E slides into 9 tissue classes, including adipose, background, debris, lymphocytes, mucus, smooth muscle, normal mucosa, stroma and cancer epithelium [74].”