A better measure of relative prediction accuracy for model selection and model estimation

By Chris Tofallis. Journal of the Operational Research Society, 2015

524 citations recorded by the source corpus.

Surveys show that the mean absolute percentage error (MAPE) is the most widely used measure of prediction accuracy in businesses and organizations. It is, however, biased: when used to select among competing prediction methods it systematically selects those whose predictions are too low. This has not been widely discussed and so is not generally known among practitioners. We explain why this happens. We investigate an alternative relative accuracy measure which avoids this bias: the log of the accuracy ratio, that is, log (prediction/actual). Relative accuracy is particularly relevant if the scatter in the data grows as the value of the variable grows (heteroscedasticity). We demonstrate using simulations that for heteroscedastic data (modelled by a multiplicative error factor) the proposed metric is far superior to MAPE for model selection. Another use for accuracy measures is in fitting parameters to prediction models. Minimum MAPE models do not predict a simple statistic and so theoretical analysis is limited. We prove that when the proposed metric is used instead, the resulting least squares regression model predicts the geometric mean. This important property allows its theoretical properties to be understood.

Subfields: Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/e47d888597262e13325bb00528cf0504e19e0499

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Bao: Making Learned Query Optimization Practical (2020, methodology): “Following [32, 42], we use Q-Error instead of relative error [63].”
Deep learning in business analytics and operations research: Models, applications and managerial implications (2018, unknown): “introducing-tensorflow-feature-columns.html , last accessed February 20, 2018. 7 We refrain from using percentage errors as seen in some Kaggle competitions: there are crisp mathematical limitations (Tofallis, 2015) and these metrics do not measure costs. Instead, we evaluate all models through the ”
Analysis of accident injury-severity outcomes: The zero-inflated hierarchical ordered probit model with correlated disturbances (2018, background): “This finding is in line with the results of the non-nested goodness-of-fit measures and may highlight the potential of the specific metric for unbiased model selection, using the relative accuracy as primary criterion of selection (Tofallis, 2015).”
Performance Metrics (Error Measures) in Machine Learning Regression, Forecasting and Prognostics: Properties and Typology (2018, methodology): “…better measure…” (Tofallis, 2015), “A new accuracy measure…” (Chen, Twycross and Garibaldi, 2017), “A new metric…” (Kim and Kim, 2016), “New Statistical Indices…” (Kyriakidis, et al, 2015), “New accuracy measures…” (Bratu, 2013), “…new methods for measuring forecast error” ( Mathai, et al, 2016).”
Comparison and combination of two NDT methods with implications for compressive strength evaluation of selected masonry and building stones (2018, methodology): “For accuracy comparison of the established regression models, the Baccuracy ratio^ metrics proposed by Tofallis (2015) was used: Accuracy Ratio ¼ ∑ lnQð Þ2 ð3Þ where Q is the ratio of the predicted to actual value.”