What accuracy statistics really measure

By Barbara A. Kitchenham, L. Pickard, Stephen G. MacDonell, Martin J. Shepperd. IEE Proceedings - Software, 2001

459 citations recorded by the source corpus.

Provides the software estimation research community with a better understanding of the meaning of, and relationship between, two statistics that are often used to assess the accuracy of predictive models: the mean magnitude relative error (MMRE) and the number of predictions within 25% of the actual, pred(25). It is demonstrated that MMRE and pred(25) are, respectively, measures of the spread and the kurtosis of the variable z, where z=estimate/actual. Thus, z is considered to be a measure of accuracy, and statistics such as MMRE and pred(25) to be measures of properties of the distribution of z. It is suggested that measures of the central location and skewness of z, as well as measures of spread and kurtosis, are necessary. Furthermore, since the distribution of z is non-normal, non-parametric measures of these properties may be needed. For this reason, box-plots of z are useful alternatives to simple summary metrics. It is also noted that the simple residuals are better behaved than the z variable, and could also be used as the basis for comparing prediction systems.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/6107a4eb8b7f887528b359ccf9733b29cf3dded2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

How Long Will It Take to Fix This Bug? (2007, methodology): “While their technique seemed to have performed favorably, unfortunately, they used magnitude of relative error for evaluation which is asymmetric [8, 6] casting doubt on the validity of the results.”
A better measure of relative prediction accuracy for model selection and model estimation (2015, methodology): “It is the use of logs that overcomes the asymmetry issue that Kitchenham et al. (2001) observed in the Q accuracy measure.”
Evaluating prediction systems in software project estimation (2012, unknown): “This was pointed out more than ten years ago by Kitchenham et al. [20] and subsequently by Foss et al. [9] and Myrtveit et al. [32].”
A Simulation Study of the Model Evaluation Criterion MMRE (2003, methodology): “Another measure akin to MRE, the magnitude of error relative to the estimate (MER), has been proposed by Kitchenham et al. [ 31 ].”
Reliability and validity in comparative studies of software prediction models (2005, background): “In fact, several recent findings directly or indirectly question the validity or reliability of our measuring procedures [15], [22], [29], [25].”
Performance Metrics (Error Measures) in Machine Learning Regression, Forecasting and Prognostics: Properties and Typology (2018, methodology): “Kitchenham et al. (2001) introduced quotient error (accuracy ratio) into software development effort forecasting industry, designating it variable z. Although this metric has been studied earlier, as an alternative to subtraction-type errors, in different environments. For example, Olver (1978) used”