Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study
By John R. Zech, Marcus A. Badgeley, Manway Liu, Anthony B. Costa, Joseph J. Titano, Eric Karl Oermann. 2018
1,601 citations recorded by the source corpus.
Background There is interest in using convolutional neural networks (CNNs) to analyze medical imaging to provide computer-aided diagnosis (CAD). Recent work has suggested that image classification CNNs may not generalize to new data as well as previously believed. We assessed how well CNNs generalized across three hospital systems for a simulated pneumonia screening task. Methods and findings A cross-sectional design with multiple model training cohorts was used to evaluate model generalizability to external sites using split-sample validation. A total of 158,323 chest radiographs were drawn from three institutions: National Institutes of Health Clinical Center (NIH; 112,120 from 30,805 patients), Mount Sinai Hospital (MSH; 42,396 from 12,904 patients), and Indiana University Network for Patient Care (IU; 3,807 from 3,683 patients). These patient populations had an age mean (SD) of 46.9 years (16.6), 63.2 years (16.5), and 49.6 years (17) with a female percentage of 43.5%, 44.8%, and 57.3%, respectively. We assessed individual models using the area under the receiver operating characteristic curve (AUC) for radiographic findings consistent with pneumonia and compared performance on different test sets with DeLong’s test. The prevalence of pneumonia was high enough at MSH (34.2%) relative to NIH and IU (1.2% and 1.0%) that merely sorting by hospital system achieved an AUC of 0.861 (95% CI 0.855–0.866) on the joint MSH–NIH dataset. Models trained on data from either NIH or MSH had equivalent performance on IU (P values 0.580 and 0.273, respectively) and inferior performance on data from each other relative to an internal test set (i.e., new data from within the hospital system used for training data; P values both <0.001). The highest internal performance was achieved by combining training and test data from MSH and NIH (AUC 0.931, 95% CI 0.927–0.936), but this model demonstrated significantly lower external performance at IU (AUC 0.815, 95% CI 0.745–0.885, P = 0.001). To test the effect of pooling data from sites with disparate pneumonia prevalence, we used stratified subsampling to generate MSH–NIH cohorts that only differed in disease prevalence between training data sites. When both training data sites had the same pneumonia prevalence, the model performed consistently on external IU data (P = 0.88). When a 10-fold difference in pneumonia rate was introduced between sites, internal test performance improved compared to the balanced model (10× MSH risk P < 0.001; 10× NIH P = 0.002), but this outperformance failed to generalize to IU (MSH 10× P < 0.001; NIH 10× P = 0.027). CNNs were able to directly detect hospital system of a radiograph for 99.95% NIH (22,050/22,062) and 99.98% MSH (8,386/8,388) radiographs. The primary limitation of our approach and the available public data is that we cannot fully assess what other factors might be contributing to hospital system–specific biases. Conclusion Pneumonia-screening CNNs achieved better internal than external performance in 3 out of 5 natural comparisons. When models were trained on pooled data from sites with different pneumonia prevalence, they performed better on new pooled data from these sites but not on external data. CNNs robustly identified hospital system and department within a hospital, which can have large differences in disease burden and may confound predictions.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Health Informatics, Radiology, Nuclear Medicine and Imaging, Medical Imaging
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Yoshua Bengio
- Quoc V. Le
- Jianfeng Gao
- Ruslan Salakhutdinov
- Sergey Levine
- Luke S. Zettlemoyer
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Bernt Schiele
- Jitendra Malik
- Ming-Hsuan Yang
- Alan L. Yuille
In Health Informatics
- Ramesh Raskar
- Praneeth Vepakomma
- Vivek Sharma
- Su-In Lee
- Scott M. Lundberg
- Stephen Seung-Yeob Baek
- Zhi Chen
- Lesia Semenova
In Radiology, Nuclear Medicine and Imaging
- P. Heng
- Thomas Liao
- Arvid Lundervold
- Qi-Yan Dou
- Lequan Yu
- Nicholas Carlini
- Praneeth Vepakomma
- Cheng Chen
In Medical Imaging
- Matthew P. Lungren
- Ben Glocker
- Konstantinos Kamnitsas
- Arvid Lundervold
- P. Heng
- Ronald M. Summers
- Qi-Yan Dou
- Cynthia Rudin
Citations in the corpus
- Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, background): “Similarly, researcher also found that current methods for chest X-ray classification are brittle even in the absence of recognized confounders [110].”
- Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “Black box models often predict the right answer for the wrong reason (the “Clever Hans” phenomenon), leading to excellent performance in training but poor performance in practice [268, 172, 225, 331, 19, 119].”
- Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead (2018, background): “[34], who noticed that their neural network was picking up on the word “portable” within an x-ray image, representing the type of x-ray equipment rather than the medical content of the image.”
- Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming (2019, background): “Examples include the failure to generalize to images with uncommon poses of objects [Alcorn et al., 2019] or to cope with small distributional changes [e.g. Zech et al., 2018, Touvron et al., 2019].”
- An overview of deep learning in medical imaging focusing on MRI (2018, background): “See [341] for a recent exploration of this issue.”
- A case-based interpretable deep learning model for classification of mass lesions in digital mammography (2021, background): “, type of equipment) rather than medical information [4, 52, 57].”
- Optimal Sparse Decision Trees (2019, neutral): “[22] J.”
- WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “In domain generalization, the training and test distributions comprise data from related but distinct domains, such as patients from different hospitals (Zech et al., 2018), images taken by different cameras (Beery et al.”
- Learning Explainable Models Using Attribution Priors (2019, background): “Previous literature about such methods has focused on the axioms they should satisfy [18, 37, 35, 6], and how attribution methods can give us insight into model behavior [19, 20, 29, 41].”
- Explainable AI under contract and tort law: legal incentives and technical challenges (2020, background): “The clinical suitability criterion is particularly important in medical contexts where the consequences of false positive or false negative outcomes may be particularly undesirable (Caruana 2015; Zech et al. 2018; Rudin and Ustun 2018).”
- Preventing Failures Due to Dataset Shift: Learning Predictive Models That Transport (2018, background): “Copyright 2019 by the author(s). deployment environment, model performance may significantly deteriorate and potentially cause harm in safety-critical applications such as healthcare (see e.g., Schulam and Saria (2017); Zech et al. (2018)).”
- Computer-Aided Diagnosis in the Era of Deep Learning (2020, methodology): “Zech et al. 67 reported large variabilities in the generalization performance of DCNNs when different combinations of training and test data collected from three clinical sites were used.”
- The Secrets of Machine Learning: Ten Things You Wish You Had Known Earlier to be More Effective at Data Analysis (2019, background): “writing indicating the type of xray equipment rather than at the actual medical content of the xray [61]; this is a bad type of confounding, and similar problems happen in almost all medical datasets and in other types of data as well.”