Data mining in metric space: an empirical analysis of supervised learning performance criteria

By R. Caruana, Alexandru Niculescu-Mizil. 2004

367 citations recorded by the source corpus.

Many criteria can be used to evaluate the performance of supervised learning. Different criteria are appropriate in different settings, and it is not always clear which criteria to use. A further complication is that learning methods that perform well on one criterion may not perform well on other criteria. For example, SVMs and boosting are designed to optimize accuracy, whereas neural nets typically optimize squared error or cross entropy. We conducted an empirical study using a variety of learning methods (SVMs, neural nets, k-nearest neighbor, bagged and boosted trees, and boosted stumps) to compare nine boolean classification performance metrics: Accuracy, Lift, F-Score, Area under the ROC Curve, Average Precision, Precision/Recall Break-Even Point, Squared Error, Cross Entropy, and Probability Calibration. Multidimensional scaling (MDS) shows that these metrics span a low dimensional manifold. The three metrics that are appropriate when predictions are interpreted as probabilities: squared error, cross entropy, and calibration, lay in one part of metric space far away from metrics that depend on the relative order of the predicted values: ROC area, average precision, break-even point, and lift. In between them fall two metrics that depend on comparing predictions to a threshold: accuracy and F-score. As expected, maximum margin methods such as SVMs and boosted trees have excellent performance on metrics like accuracy, but perform poorly on probability metrics such as squared error. What was not expected was that the margin methods have excellent performance on ordering metrics such as ROC area and average precision. We introduce a new metric, SAR, that combines squared error, accuracy, and ROC area into one metric. MDS and correlation analysis shows that SAR is centrally located and correlates well with other metrics, suggesting that it is a good general purpose metric to use when more specific criteria are not known.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Information Systems

Citations in the corpus

Learning Optimized Risk Scores (2016, neutral): “[21] Caruana, Rich and Alexandru Niculescu-Mizil.”
An experimental comparison of performance measures for classification (2009, methodology): “The problem of using 100 as [6] suggest is that it might be a much too large bin for small datasets.”
On the effectiveness of preprocessing methods when dealing with different levels of class imbalance (2012, result): “In a similar way to the work by Caruana and Niculescu-Mizil [51], for each metric, we built a 25 D table, where D denotes the number of databases.”
Optimized Scoring Systems: Toward Trust in Machine Learning for Healthcare and Criminal Justice (2018, methodology): “Logistic regression produces risk-calibrated models (12, 61) but when rounding or other post-processing steps are done to a logistic regression model, it can drastically alter calibration.”
A Comprehensive Survey of Data Mining-based Fraud Detection Research (2010, neutral): “Caruana and Niculescu-Mizil (2004) argues that the most effective way to assess supervised algorithms is to use one metric from threshold, ordering, and probability metrics; and they justify using the average of mean squared error, accuracy, and AUC.”
Error Metrics and Performance Fitness Indicators for Artificial Intelligence and Machine Learning in Engineering and Sciences (2020, methodology): “CAL is calculated as the mean of these binned calibration errors [102].”
A support vector method for multivariate performance measures (2005, background): “…many different variants of convenient and tractable performance measures, aiming to find one that performs well for the application specific performance measure after post-processing the resulting model (e.g. (Lewis, 2001; Yang, 2001; Abe et al., 2004; Caruana & Niculescu-Mizil, 2004)).”
Hinge Rank Loss and the Area Under the ROC Curve (2007, methodology): “This is supported by our experiments in Section 7 as well as by many experimental findings in the literature [3,4,5,6], as discussed in Section 8.”
Supervised classification and mathematical optimization (2013, methodology): “See [52, 196] for different proposals and an empirical comparison of different criteria.”
Measuring the accuracy of species distribution models: a review (2009, methodology): “1 Caruana and Niculescu-Mizil (2004) Coefficient of determination”
Improved empirical methods in reinforcement-learning evaluation (2015, methodology): “In classification, just to name a few, the metrics used could be: Accuracy, Precision, Recall, Area Under Curve and Receiver Operating Characteristics (Caruana and Niculescu-Mizil, 2004).”
Measuring the relative effect of factors affecting species distribution model predictions (2014, background): “We focused on the root-mean-squared error (RMSE; Caruana & Niculescu-Mizil 2004; Liu, White & Newell 2011) owing to its advantages in our context: it compares probabilities to probabilities, it is easily interpretable, and the distribution of log(RMSE) was found to be appropriate for the analysis…”
Which method predicts recidivism best?: a comparison of statistical, machine learning and data mining predictive models (2013, background): “(d) SAR: the deﬁnition of SAR (Caruana and Niculescu-Mizil, 2004) is SAR = .”
Forecasting cancellation rates for services booking revenue management using data mining (2010, background): “Recent studies suggest that random forest is among the most competitive probability estimation method (Caruana and Niculescu-Mizil 2004, 2006).”