The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets
By Takaya Saito, Marc Rehmsmeier. 2015
4,380 citations recorded by the source corpus.
Binary classifiers are routinely evaluated with performance measures such as sensitivity and specificity, and performance is frequently illustrated with Receiver Operating Characteristics (ROC) plots. Alternative measures such as positive predictive value (PPV) and the associated Precision/Recall (PRC) plots are used less frequently. Many bioinformatics studies develop and evaluate classifiers that are to be applied to strongly imbalanced datasets in which the number of negatives outweighs the number of positives significantly. While ROC plots are visually appealing and provide an overview of a classifier's performance across a wide range of specificities, one can ask whether ROC plots could be misleading when applied in imbalanced classification scenarios. We show here that the visual interpretability of ROC plots in the context of imbalanced datasets can be deceptive with respect to conclusions about the reliability of classification performance, owing to an intuitive but wrong interpretation of specificity. PRC plots, on the other hand, can provide the viewer with an accurate prediction of future classification performance due to the fact that they evaluate the fraction of true positives among positive predictions. Our findings have potential implications for the interpretation of a large number of studies that use ROC plots on imbalanced datasets.
Subfields: Accounting, Artificial Intelligence, Information Systems, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Accounting
- Daniel M. Low
- Jung-un Baek
- Isak Johansson-Åkhe
- Satrajit S. Ghosh
- Daniel Quang
- Carlos Riquelme
- Xiaohui S. Xie
- Kevin Gimpel
In Artificial Intelligence
In Information Systems
- Daniel M. Low
- Jung-un Baek
- Isak Johansson-Åkhe
- Jan Grau
- Dawn Song
- Rayadurgam Srikant
- Sarah Graham
- Emma Pierson
In Evaluation & Benchmarks
- Dawn Song
- Jacob Steinhardt
- Steven Basart
- Daniel M. Low
- Jung-un Baek
- Isak Johansson-Åkhe
- Jan Grau
- Rayadurgam Srikant
Citations in the corpus
- Natural Adversarial Examples (2019, methodology): “With these anomaly scores, we can compute the area under the precision-recall curve [43].”
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks (2016, methodology): “The baseline detector has an AUPR approximately equal to the precision (Saito & Rehmsmeier, 2015), and a “perfect” classifier has an AUPR of 100% .”
- Early Methods for Detecting Adversarial Images (2016, methodology): “The baseline detector has an AUPR equal to the precision (Saito & Rehmsmeier, 2015), and a “perfect” classifier has an AUPR of 100% .”
- Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks (2017, background): “(4) AUPR is the Area under the Precision-Recall curve, which is another threshold independent metric (Manning et al., 1999; Saito & Rehmsmeier, 2015).”
- Coarse race data conceals disparities in clinical risk score performance (2023, background): “For example, the baseline AUPRC for a random classifier is p ( y = 1) (Saito and Rehmsmeier, 2015), so AUPRC will naturally tend to be higher for groups with higher outcome frequency.”
- IDRiD: Diabetic Retinopathy - Segmentation and Grading Challenge. (2020, methodology): “As in the lesion segmentation task(s) background overwhelms foreground, a highly imbalanced scenario, the performance of this task was measured using area under precision ( a.k.a. Positive Predictive Value (PPV)) recall ( a.k.a. Sensitivity (SN)) curve (AUPR) ( Saito and Rehmsmeier, 2015 ).”
- Role of Artificial Intelligence in Patient Safety Outcomes: Systematic Literature Review (2020, background): “Another common measure, the area under the receiver operating characteristic curve (AUROC), is also not necessarily the best metric for clinical applicability [27].”
- Predicting diabetes-related hospitalizations based on electronic health records (2018, background): “[ 33] While ROC curves could provide misleading interpretation of specificity when utilized in imbalanced classification cases,[34, 35] (i.e., when one class has many more samples than the other), the PRC presents a more accurate measurement of imbalanced classification performance because it also…”
- Artificial Intelligence for Mental Health and Mental Illnesses: An Overview (2019, background): “Precision is the proportion of positive identifications (e.g., presence of MDD) that are correctly classified by the algorithm (TPs/(TPs + FPs)) [86, 91].”
- SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation (2022, methodology): “But as pointed out in [11,13,37], in imbalanced dataset where performance of minor class is more important, AU-ROC might provide an inflated view of performance which may cause challenges in measuring models’ true capabilities.”
- Evaluating Merging Strategies for Sampling-based Uncertainty Techniques in Object Detection (2018, background): “The spatial cost can be calculated as the negation of IoU, PAC or EAC between detections and a cluster.”
- The area under the precision‐recall curve as a performance metric for rare binary events (2019, background): “Precision- recall curves have also been recognized as useful for classification performance assessment for unbalanced binary responses in bioinformatics (Saito & Rehmsmeier, 2015).”
- A Clinically Applicable Approach to Continuous Prediction of Future Acute Kidney Injury (2019, neutral): “PR AUC is known to be more informative for class-imbalanced predictive tasks 53 , as it is more sensitive to changes in the number of false positive predictions.”
- 3D Cell Nuclear Morphology: Microscopy Imaging Dataset And Voxel-Based Morphometry Classification Results (2017, methodology): “Given the class imbalance in L2OGO, we used AUC, AUPR, and F1 score to compare algorithms [20].”