Patch-based Convolutional Neural Network for Whole Slide Tissue Image Classification

By Le Hou, Dimitris Samaras, Tahsin M. Kurç, Yi Gao, James E. Davis, Joel H. Saltz. Computer Vision and Pattern Recognition, 2015

873 citations recorded by the source corpus.

Convolutional Neural Networks (CNN) are state-of-theart models for many image classification tasks. However, to recognize cancer subtypes automatically, training a CNN on gigapixel resolution Whole Slide Tissue Images (WSI) is currently computationally impossible. The differentiation of cancer subtypes is based on cellular-level visual features observed on image patch scale. Therefore, we argue that in this situation, training a patch-level classifier on image patches will perform better than or similar to an image-level classifier. The challenge becomes how to intelligently combine patch-level classification results and model the fact that not all patches will be discriminative. We propose to train a decision fusion model to aggregate patch-level predictions given by patch-level CNNs, which to the best of our knowledge has not been shown before. Furthermore, we formulate a novel Expectation-Maximization (EM) based method that automatically locates discriminative patches robustly by utilizing the spatial relationships of patches. We apply our method to the classification of glioma and non-small-cell lung carcinoma cases into subtypes. The classification accuracy of our method is similar to the inter-observer agreement between pathologists. Although it is impossible to train CNNs on WSIs, we experimentally demonstrate using a comparable non-cancer dataset of smaller images that a patch-based CNN can outperform an image-based CNN.

Subfields: Artificial Intelligence, Biophysics, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/004ef85f9c53dc0db6dc3adc1ca8ecb71f7a06b4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Artificial intelligence and computational pathology (2021, background): “The development of bright ﬁ eld and ﬂ uorescent slide scanners made possible the virtualizing and digitalizing the whole glass slides [8].”
Disease prediction using graph convolutional networks: Application to Autism Spectrum Disorder and Alzheimer's disease (2018, background): “…(e.g. regular pixel grid) to yield hierarchies of features, has led to their application in numerous diﬀerent problems related to 2D and 3D images, like image segmentation (Havaei et al., 2017) and classiﬁcation (Hou et al., 2016), long before their recent re-emergence (Sahiner et al., 1996).”
Methods for Segmentation and Classification of Digital Microscopy Tissue Images (2018, background): “, (Hou et al. 2016) trained a patch-level classifier to classify glioma and NSCLC WSIs into different cancer types.”
Multiple Instance Learning for Heterogeneous Images: Training a CNN for Histopathology (2018, methodology): “In the latter, MI aggregation is incorporated while training the bag classifier as in other MI methods [1,4].”
Near real-time intraoperative brain tumor diagnosis using stimulated Raman histology and deep neural networks (2020, neutral): “…the fact that normal brain tissue and nondiagnostic regions have similar histologic features among all patients, resulting in high patch-level classification accuracy for normal brain, and eliminating the need to train an additional classifier based on the patch-level probability histograms 39 .”
Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning (2020, methodology): “, a convolutional neural network (CNN) [20, 53, 31, 10, 29].”
Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning (2022, neutral): “- subtype, grade, stage, survival, origin) [12, 19, 37, 38, 52, 53, 68, 70, 85].”
Deep learning with multimodal representation for pancancer prognosis prediction (2019, background): “In the 74 related field of tumor classification from WSIs, a ”decision-fusion” model that randomly 75 samples patches and integrates them into a Gaussian mixture has yielded accurate 76 predictions [25].”
DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification (2022, background): “MIL models generally can be divided into two groups, based on whether the ﬁnal bag predictions are directly from instance predictions [3, 9, 12, 15, 17, 47], or from the aggregations of features of instance [14, 18, 23, 34, 35, 42, 53].”
The impact of pre- and post-image processing techniques on deep learning frameworks: A comprehensive review for digital pathology image analysis (2020, methodology): “Several studies demonstrated that aggregating patch-level CNN predictions for WSI classification significantly out-performs patch-level CNNs with major voting [73,109].”
Predicting cancer outcomes from histology and genomics using convolutional networks (2017, methodology): “…of human experts using the current clinical standard for classifying brain tumors and presents an innovative approach for objective, accurate, and integrated prediction of patient outcomes. challenges, CNNs have been successfully used extensively for medical image analysis (9, 11, 16 – 26).”
Artificial Intelligence and Digital Pathology: Challenges and Opportunities (2018, neutral): “Of note, patch‐based ANNs have been shown to outperform image‐based ANNs.[20]”
Segment convolutional neural networks (Seg-CNNs) for classifying relations in clinical notes (2018, background): “Recent studies on applying convolutional neural networks (CNNs) to clinical datasets aimed to automatically learn feature representations to reduce the need for engineered features and have achieved some success on specific tasks, such as medical image analysis.(29) Most recently, Sahu et al.”
Unsupervised domain adaptation for medical imaging segmentation with self-ensembling (2018, methodology): “Though patch-wise methods continue to be actively researched (Hou et al., 2016) and have led to several advances in segmentation (Lai, 2015), presently, the most common deep architecture for segmentation is or is based on the so-called Fully Convolutional Network (FCN) (Long et al., 2015).”
Pathologist-level interpretable whole-slide cancer diagnosis with deep learning (2019, background): “Previous work in pathology diagnosis only classified small tissue images 7,9,10 , used poorly generalizable image-processing techniques 11 or lacked large-scale validation 12,13 .”
Weakly Supervised Deep Learning for Whole Slide Lung Cancer Image Analysis (2020, methodology): “Therefore, it would be appealing to train the cancer region detector with a minimum annotation (e.g., image-level labels) [35], [36], [38] which can be more easily acquired in clinical practice.”
Spatial Organization and Molecular Correlation of Tumor-Infiltrating Lymphocytes Using Deep Learning on Pathology Images (2018, positive): “One can train the unsupervised autoencoder on image patches with the object to be classiﬁed (e.g., nucleus) in the center of each patch (Hou et al., 2016a; Murthy et al., 2017) in order to capture the visual variance of the object more accurately.”