CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison

By Jeremy Irvin, Pranav Rajpurkar, Pranav Rajpurkar, M. Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, C. Chute, Henrik Marklund, Behzad Haghgoo and 12 more. 2019

3,612 citations recorded by the source corpus.

Large, labeled datasets have driven deep learning methods to achieve expert-level performance on a variety of medical imaging tasks. We present CheXpert, a large dataset that contains 224,316 chest radiographs of 65,240 patients. We design a labeler to automatically detect the presence of 14 observations in radiology reports, capturing uncertainties inherent in radiograph interpretation. We investigate different approaches to using the uncertainty labels for training convolutional neural networks that output the probability of these observations given the available frontal and lateral radiographs. On a validation set of 200 chest radiographic studies which were manually annotated by 3 board-certified radiologists, we find that different uncertainty approaches are useful for different pathologies. We then evaluate our best model on a test set composed of 500 chest radiographic studies annotated by a consensus of 5 board-certified radiologists, and compare the performance of our model to that of 3 additional radiologists in the detection of 5 selected pathologies. On Cardiomegaly, Edema, and Pleural Effusion, the model ROC and PR curves lie above all 3 radiologist operating points. We release the dataset to the public as a standard benchmark to evaluate performance of chest radiograph interpretation models.

Subfields: Computer Vision and Pattern Recognition, Pulmonary and Respiratory Medicine, Radiology, Nuclear Medicine and Imaging, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Pulmonary and Respiratory Medicine

In Radiology, Nuclear Medicine and Imaging

In Evaluation & Benchmarks

Citations in the corpus

Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, methodology): “Since a widely-used annotated benchmark for this setting is not available, we create our own dataset by re-using existing annotations in the CheXpert dataset (Irvin et al., 2019) and additional expert annotations from a board-certiﬁed radiologist.”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, background): “While machine learning methods, artificial neural networks, brain-machine interfaces and related subfields have recently demonstrated promising performance in performing medical tasks, they are hardly perfect [1] [2] [3] [4] [5] [6] [7] [8].”
What is being transferred in transfer learning? (2020, methodology): “We use CheXpert [21] which is a medical imaging dataset of chest x-rays considering different diseases.”
Generative Adversarial Network in Medical Imaging: A Review (2018, methodology): “With the advent of large volume labeled datasets, such as the CheXpert (Irvin et al., 2019), it seems there is diminishing return in the employment of GANs for image generation, especially for classification.”
Identifying pneumonia in chest X-rays: A deep learning approach (2019, background): “Irvin et al. [20] stated that large labeled dataset is the key to success for prediction and classiﬁcation tasks.”
Big Self-Supervised Models Advance Medical Image Classification (2021, methodology): “CheXpert [23] is a large open source dataset of de-identified chest radiograph (X-ray) images.”
MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs (2019, methodology): “Images are also provided with one or more labels derived from the corresponding free-text radiology report using open source labelers [12, 13].”
Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis (2020, methodology): “The uncertainty labels included in CheXpert were interpreted as negative following the U-Zeros approach discussed in the original paper (30).”
AI in health and medicine (2022, background): “Labels can also be applied by other AI models, as in some weak-supervision setups 95 , but these labels again carry the risk of noise.”
COVID-19 Image Data Collection: Prospective Predictions Are the Future (2020, methodology): “, 2019]), neural networks have even been able to achieve performance near radiologist levels [Rajpurkar et al., 2018, 2017; Irvin et al., 2019; Putha et al., 2018a; Majkowska et al., 2019; Putha et al., 2018b].”
PubMedCLIP: How Much Does CLIP Benefit Visual Question Answering in the Medical Domain? (2023, background): “Previous work (Rajpurkar et al., 2017; Wang et al., 2017; Irvin et al., 2019; Johnson et al., 2019a) also proposes large-scale multi-modal datasets in the medical domain.”
A Multi-Organ Nucleus Segmentation Challenge (2020, background): “Similarly, CheXpert [37], which is one of the largest medical image segmentation datasets has only 224, 316 images.”
Hidden stratification causes clinically meaningful failures in machine learning for medical imaging (2019, background): “As a result, medical imagery analysts have commonly begun either to use semi-automated labelling techniques [34, 14, 18, 10, 14] or to apply human expertise to produce a narrow or incomplete set of visual labels [25] rather than exhaustively labelling all possible findings and variations.”
Federated learning for predicting clinical outcomes in patients with COVID-19 (2021, methodology): “9a, the model fuses information from both EMR and CXR features (based on a modified ResNet34 with spatial attention 66 pretrained on the CheXpert dataset) 67 and the Deep & Cross network 68 .”
Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review (2020, neutral): “• Image: MNIST [194], EMNIST [60], CelebA [200], CheXpert [152], ImageNet [77], ISIC Skin Lesion [59], ADNI [225], ChestX-ray8 [326].”
Self-supervised Cross-silo Federated Neural Architecture Search (2021, methodology): “In this subsection, we introduce two benchmark datasets that are used for our evaluation, ModelNet40 [64] and CheXpert-14 [65], with tailored modifications to mimic the real-world data distribution.”
Preparing Medical Imaging Data for Machine Learning (2020, background): “Moreover, many open-source data sets are restricted to noncommercial (research only) use (79).”