An experimental comparison of performance measures for classification

By Cèsar Ferri, JosÃ© Hernandez-Orallo, R. Modroiu. Pattern Recognition Letters, 2009

873 citations recorded by the source corpus.

Performance metrics in classification are fundamental in assessing the quality of learning methods and learned models. However, many different measures have been defined in the literature with the aim of making better choices in general or for a specific application area. Choices made by one metric are claimed to be different from choices made by other metrics. In this work, we analyse experimentally the behaviour of 18 different performance metrics in several scenarios, identifying clusters and relationships between measures. We also perform a sensitivity analysis for all of them in terms of several traits: class threshold choice, separability/ranking quality, calibration performance and sensitivity to changes in prior class distribution. From the definitions and experiments, we make a comprehensive analysis of the relationships between metrics, and a taxonomy and arrangement of them according to the previous traits. This can be useful for choosing the most adequate measure (or set of measures) for a specific application. Additionally, the study also highlights some niches in which new measures might be defined and also shows that some supposedly innovative measures make the same choices (or almost) as existing ones. Finally, this work can also be used as a reference for comparing experimental results in pattern recognition and machine learning literature, when using different measures.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/41b0fbe7b579b04e0f2de087f650d85b2a071c5a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Facing Imbalanced Data Recommendations for the Use of Performance Metrics (2013, background): “When base rate is low, high accuracy can result even when alternative methods rarely if ever agree [12], [14].”
On the Use of the Adjusted Rand Index as a Metric for Evaluating Supervised Classification (2009, background): “For a detailed comparison of performance measures for classification please refer to [6].”
An overview of ensemble methods for binary classifiers in multi-class problems: Experimental study on one-vs-one and one-vs-all schemes (2011, background): “However, in the literature most of the performance measures are designed only for two-class problems [69] [27].”
Re-evaluating evaluation (2018, background): “A comprehensive study of performance metrics for machine learning and AI can be found in [31–35].”
A Survey of Predictive Modeling on Imbalanced Domains (2016, methodology): “…et al. [2006], Wu et al. [2007], Weng and Poon [2008], Garc´ıa et al. [2008], Batuwita and Palade [2009], Garc´ıa et al. [2009, 2010], Hand [2009], Ferri et al. [2009], Sokolova and Lapalme [2009], Thai-Nghe et al. [2011], Ferri et al. [2011a], and Batuwita and Palade [2012] Regression (3.2)…”
Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement (2017, background): “…(Drummond and Japkowicz, 2010; Japkowicz and Shah, 2011), follows the general form of equation 3 where M is the repository, p is the choice of datasets and R is one particular performance metric (accuracy, AUC, Brier score, F-measure, MSE, etc., Ferri et al, 2009; Hernández-Orallo et al, 2012a).”
Landslide susceptibility mapping using GIS-based multi-criteria decision analysis, support vector machines, and logistic regression (2014, methodology): “On the other hand, several statistical tests including F-measure, mean squared error, and ROC statistics have been suggested to compare performances of the methods (De Leeuw et al. 2006; Ferri et al. 2009).”
A Comparison of MCC and CEN Error Measures in Multi-Class Prediction (2010, background): “…classiﬁcation is still an open research topic, although several functions have been proposed in the last few years: see (Sokolova & Lapalme, 2009; Ferri et al., 2009) for two comparing reviews, (Felkin, 2007) for a discussion of the di ﬀ erences between the use of the same classiﬁer on a…”
Probabilistic Extension of Precision, Recall, and F1 Score for More Thorough Evaluation of Classification Models (2020, methodology): “In this document we will use the taxonomy introduced by Ferri et al. (2009), classifying metrics into 3 categories: • Threshold-based metrics, using a qualitative understanding of error, such as Accuracy, Precision, and Recall.”
To tune or not to tune the number of trees in random forest? (2017, methodology): “Following Hand and Till (2001), the AUC can also be generalized to the multiclass case as where AUC( j, k ) is the AUC between class k and j , see also Ferri et al. (2009) for more details.”
Reliable Crowdsourcing and Deep Locality-Preserving Learning for Unconstrained Facial Expression Recognition (2019, methodology): “Because real-world expressions have an imbalanced distribution, the accuracy metric, which is employed as the evaluation criterion in most datasets, is not used in RAF as it is especially sensitive to bias and is not effective for imbalanced data [21].”
Hyperparameters and tuning strategies for random forest (2018, background): “Deﬁnitions and a good comparison between measures for classiﬁcation can be found in Ferri et al. (2009).”
Spatiotemporal modeling and prediction in cellular networks: A big data enabled deep learning approach (2017, methodology): “These two baselines were compared with the proposed model in terms of three commonly used performance metrics [10]: Mean Squared Error (MSE), Mean Absolute Error (MAE), and Log Loss (also known as binary cross-entropy).”