Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists

By Pranav Rajpurkar, Jeremy Irvin, Robyn L. Ball, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Yi Ding, Aarti Bagul, Curtis P. Langlotz and 11 more. 2018

1,270 citations recorded by the source corpus.

Background Chest radiograph interpretation is critical for the detection of thoracic diseases, including tuberculosis and lung cancer, which affect millions of people worldwide each year. This time-consuming task typically requires expert radiologists to read the images, leading to fatigue-based diagnostic error and lack of diagnostic expertise in areas of the world where radiologists are not available. Recently, deep learning approaches have been able to achieve expert-level performance in medical image interpretation tasks, powered by large network architectures and fueled by the emergence of large labeled datasets. The purpose of this study is to investigate the performance of a deep learning algorithm on the detection of pathologies in chest radiographs compared with practicing radiologists. Methods and findings We developed CheXNeXt, a convolutional neural network to concurrently detect the presence of 14 different pathologies, including pneumonia, pleural effusion, pulmonary masses, and nodules in frontal-view chest radiographs. CheXNeXt was trained and internally validated on the ChestX-ray8 dataset, with a held-out validation set consisting of 420 images, sampled to contain at least 50 cases of each of the original pathology labels. On this validation set, the majority vote of a panel of 3 board-certified cardiothoracic specialist radiologists served as reference standard. We compared CheXNeXt’s discriminative performance on the validation set to the performance of 9 radiologists using the area under the receiver operating characteristic curve (AUC). The radiologists included 6 board-certified radiologists (average experience 12 years, range 4–28 years) and 3 senior radiology residents, from 3 academic institutions. We found that CheXNeXt achieved radiologist-level performance on 11 pathologies and did not achieve radiologist-level performance on 3 pathologies. The radiologists achieved statistically significantly higher AUC performance on cardiomegaly, emphysema, and hiatal hernia, with AUCs of 0.888 (95% confidence interval [CI] 0.863–0.910), 0.911 (95% CI 0.866–0.947), and 0.985 (95% CI 0.974–0.991), respectively, whereas CheXNeXt’s AUCs were 0.831 (95% CI 0.790–0.870), 0.704 (95% CI 0.567–0.833), and 0.851 (95% CI 0.785–0.909), respectively. CheXNeXt performed better than radiologists in detecting atelectasis, with an AUC of 0.862 (95% CI 0.825–0.895), statistically significantly higher than radiologists' AUC of 0.808 (95% CI 0.777–0.838); there were no statistically significant differences in AUCs for the other 10 pathologies. The average time to interpret the 420 images in the validation set was substantially longer for the radiologists (240 minutes) than for CheXNeXt (1.5 minutes). The main limitations of our study are that neither CheXNeXt nor the radiologists were permitted to use patient history or review prior examinations and that evaluation was limited to a dataset from a single institution. Conclusions In this study, we developed and validated a deep learning algorithm that classified clinically important abnormalities in chest radiographs at a performance level comparable to practicing radiologists. Once tested prospectively in clinical settings, the algorithm could have the potential to expand patient access to chest radiograph diagnostics.

Subfields: Computer Vision and Pattern Recognition, Pulmonary and Respiratory Medicine, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Pulmonary and Respiratory Medicine

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, background): “…(Wang and Wong, 2020), which involves multi-class chest image classiﬁcation into three categories ( COVID19 , non-COVID pneumonia or normal ); and 4) MURA bony abnormality detection (Rajpurkar et al., 2018a), which involves binary classiﬁcation of a musculoskeletal image into abnormal or normal .”
COVID-19 Image Data Collection: Prospective Predictions Are the Future (2020, background): “…et al., 2017], PadChest [Bustos et al., 2019] or MIMIC-CXR [Johnson et al., 2019]), neural networks have even reported being able to achieve performance near radiologist levels [Rajpurkar et al., 2018, 2017; Irvin et al., 2019; Putha et al., 2018a; Majkowska et al., 2019; Putha et al., 2018b].”
Artificial Intelligence Versus Clinicians in Disease Diagnosis: Systematic Review (2019, background): “Such AI-assisted imaging-related clinical tasks can reduce the cognitive burden on human experts [17] and thus increase the efficiency of health care delivery.”
VinDr-CXR: An open dataset of chest X-rays with radiologist’s annotations (2020, methodology): “However, it is costly and time-consuming to build such datasets due to several constraints: (1) medical data are hard to retrieve from hospitals or medical centers; (2) physician’s time is precious; (3) the annotation of medical images requires a consensus of several expert readers to overcome human”
Hidden stratification causes clinically meaningful failures in machine learning for medical imaging (2019, methodology): “labeled for one of 14 different thoracic pathologies. In our analysis, we leverage a pretrained Densenet-121 model provided by Zech [39] which reproduces the procedure and results of Rajpurkar et al. [26] on this dataset. During error auditing, where examples of false positive and false negative pre”
Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension (2020, positive): “CONSORT-AI item 19, which requires specification of any plans to analyse performance errors, was added to emphasise the importance of anticipating systematic errors made by the algorithm and their consequences.”
Shifting machine learning for healthcare from development to deployment and from models to data (2022, background): “In the past decade, many deep-learning models for medical-image classification 3,37 , segmentation 38,39 and detection 40 have achieved physician-level performance.”
Deep learning approaches for COVID-19 detection based on chest X-ray images (2020, positive): “Rajpurkar et al. (2018) developed a 121-layered CNN architecture named CheXNeXt for the classification of 14 different pathologies based on chest X-ray images.”