The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets

By Takaya Saito, Marc Rehmsmeier. 2015

4,380 citations recorded by the source corpus.

Binary classifiers are routinely evaluated with performance measures such as sensitivity and specificity, and performance is frequently illustrated with Receiver Operating Characteristics (ROC) plots. Alternative measures such as positive predictive value (PPV) and the associated Precision/Recall (PRC) plots are used less frequently. Many bioinformatics studies develop and evaluate classifiers that are to be applied to strongly imbalanced datasets in which the number of negatives outweighs the number of positives significantly. While ROC plots are visually appealing and provide an overview of a classifier's performance across a wide range of specificities, one can ask whether ROC plots could be misleading when applied in imbalanced classification scenarios. We show here that the visual interpretability of ROC plots in the context of imbalanced datasets can be deceptive with respect to conclusions about the reliability of classification performance, owing to an intuitive but wrong interpretation of specificity. PRC plots, on the other hand, can provide the viewer with an accurate prediction of future classification performance due to the fact that they evaluate the fraction of true positives among positive predictions. Our findings have potential implications for the interpretation of a large number of studies that use ROC plots on imbalanced datasets.

Subfields: Accounting, Artificial Intelligence, Information Systems, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Accounting

  1. Daniel M. Low
  2. Jung-un Baek
  3. Isak Johansson-Åkhe
  4. Satrajit S. Ghosh
  5. Daniel Quang
  6. Carlos Riquelme
  7. Xiaohui S. Xie
  8. Kevin Gimpel

In Artificial Intelligence

  1. Percy Liang
  2. Quoc V. Le
  3. Dawn Song
  4. Mohit Bansal
  5. Yixuan Li
  6. Yoshua Bengio
  7. Jianfeng Gao
  8. Sergey Levine

In Information Systems

  1. Daniel M. Low
  2. Jung-un Baek
  3. Isak Johansson-Åkhe
  4. Jan Grau
  5. Dawn Song
  6. Rayadurgam Srikant
  7. Sarah Graham
  8. Emma Pierson

In Evaluation & Benchmarks

  1. Dawn Song
  2. Jacob Steinhardt
  3. Steven Basart
  4. Daniel M. Low
  5. Jung-un Baek
  6. Isak Johansson-Åkhe
  7. Jan Grau
  8. Rayadurgam Srikant

Citations in the corpus