Unachievable Region in Precision-Recall Space and Its Effect on Empirical Evaluation
By Kendrick Boyd, Jesse Davis, David Page, Vítor Santos Costa. 2012
121 citations recorded by the source corpus.
Precision-recall (PR) curves and the areas under them are widely used to summarize machine learning results, especially for data sets exhibiting class skew. They are often used analogously to ROC curves and the area under ROC curves. It is known that PR curves vary as class skew changes. What was not recognized before this paper is that there is a region of PR space that is completely unachievable, and the size of this region depends only on the skew. This paper precisely characterizes the size of that region and discusses its implications for empirical evaluation methodology in machine learning.
Subfields: Artificial Intelligence, Control and Systems Engineering, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Neil Houlsby
- Xiaohua Zhai
- Mario Lucic
- Quoc V. Le
- Percy Liang
- Katherine A. Heller
- Ghassen Jerfel
- Xuezhi Wang
In Control and Systems Engineering
- Andrea Montanari
- Katherine A. Heller
- Xiaohua Zhai
- Neil Houlsby
- Matthew D. Hoffman
- Mario Lucic
- Peter A. Flach
- Meelis Kull
In Evaluation & Benchmarks
- Neil Houlsby
- Xiaohua Zhai
- Jacob Eisenstein
- Andrea Montanari
- Katherine A. Heller
- David Sculley
- Kellie Webster
- Xuezhi Wang
Citations in the corpus
- Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, methodology): “Our ensemble of predictors achieves tightly constrained normalized PRAUC values between 34.59 and 36.61.”
- Precision-Recall-Gain Curves: PR Analysis Done Right (2015, background): “Furthermore, PR plots have unachievable regions at the lower right-hand side, the size of which depends on the class distribution [1].”