Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
By David Martin Ward Powers. 2011
6,774 citations recorded by the source corpus.
Commonly used evaluation measures including Recall, Precision, F-Factor and Rand Accuracy are biased and should not be used without clear understanding of the biases, and corresponding identification of chance or base case levels of the statistic. Using these measures a system that performs worse in the objective sense of Informedness, can appear to perform better under any of these commonly used measures. We discuss several concepts and measures that reflect the probability that prediction is informed versus chance. Informedness and introduce Markedness as a dual measure for the probability that prediction is marked versus chance. Finally we demonstrate elegant connections between the concepts of Informedness, Markedness, Correlation and Significance as well as their intuitive relationships with Recall and Precision, and outline the extension from the dichotomous case to the general multi-class case. .
Subfields: Management Information Systems
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Management Information Systems
Citations in the corpus
- Fairness in Criminal Justice Risk Assessments: The State of the Art (2017, background): “Powers (2011) provides an excellent review.”
- SoK: Security and Privacy in Machine Learning (2018, background): “Integrity is essential in ML, and is the center of attention for most performance metrics used: e.g., accuracy [31].”
- Passban IDS: An Intelligent Anomaly-Based Intrusion Detection System for IoT Edge Devices (2020, background): “In order to assess the performance of Passban from a pure machine learning perspective, we start from the classical terms of a confusion matrix [67], namely, true (TP), true negatives (TNs), FP, and false negatives (FN), recalling that, in our experiments, a TP occurs when Passban correctly detects…”
- Earthquake Phase Association Using a Bayesian Gaussian Mixture Model (2021, methodology): “Here, we define the associated event that is within 3s and 15 km from the true event as true positive association to calculate precision, recall, and F1-score (Powers, 2020).”
- Combining RNA-seq data and homology-based gene prediction for plants, animals and fungi (2017, background): “In machine learning, recall, also known as sensitivity, and precision, which is called specificity in the context of gene prediction evaluation [31], are combined into a single scalar value called F1 measure [32] that can be compared more easily.”
- The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation (2021, methodology): “BA and BM were developed independently by Brodersen and Powers [22, 54].”
- Generalized Out-of-Distribution Detection: A Survey (2021, neutral): “Note that there can be two variants of AUPR values: one treating “normal” as the positive class, and the other treating “abnormal” as the positive class.”
- Credit Card Fraud Detection Using AdaBoost and Majority Voting (2019, positive): “[25] to handle data imbalance problems.”
- The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation (2020, methodology): “Since analyzing all the four categories of the confusion matrix separately would be time-consuming, statisticians introduced some useful statistical rates able to immediately describe the quality of a prediction [22], aimed at conveying into a single figure the structure of M.”
- A review of epileptic seizure detection using machine learning classifiers (2020, neutral): “Except for the accuracy, the performance of the classifiers is commonly measured by the following metrics such as precision, recall, and f-measure [58].”
- Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels (2019, methodology): “Following the standard metrics (Powers, 2011), we measure the identification performance in terms of Label Precision (LP ) (Han et al.”