A Simulation Study of the Model Evaluation Criterion MMRE

By T. Foss, Erik Stensrud, Barbara A. Kitchenham, Ingunn Myrtveit. IEEE Trans. Software Eng., 2003

499 citations recorded by the source corpus.

Subfields: Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/6e163355c8e6de3c293e32946ce42ba125466432

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

Citations in the corpus

A Systematic Review of Software Development Cost Estimation Studies (2007, background): “This general picture served as a starting point for deeper investigation of findings that seemed, from the authors’ perspectives, to suggest important shortcomings in estimation research and possibilities for improvement.2”
Performance Metrics (Error Measures) in Machine Learning Regression, Forecasting and Prognostics: Properties and Typology (2018, background): “Foss et al (2003) concluded that MMRE (MAPE), another very popular metric, is “an unreliable selection criterion and may have misled the entire software engineering discipline.””
Cross versus Within-Company Cost Estimation Studies: A Systematic Review (2007, methodology): “We have to rely on accuracy statistics, which are known to be biased [5].”
Evaluating prediction systems in software project estimation (2012, unknown): “This was pointed out more than ten years ago by Kitchenham et al. [20] and subsequently by Foss et al. [9] and Myrtveit et al. [32].”
Reliability and validity in comparative studies of software prediction models (2005, methodology): “[15] demonstrated that MMRE is an invalid measure for selecting between two competing, linear prediction models, thus refuting claim 3.”
A better measure of relative prediction accuracy for model selection and model estimation (2015, methodology): “The third column of the table gives the performance of another metric which was found to perform very well by Foss et al. (2003) in that it correctly chose the correct model more often than the other criteria they studied.”
How Long Will It Take to Fix This Bug? (2007, methodology): “While their technique seemed to have performed favorably, unfortunately, they used magnitude of relative error for evaluation which is asymmetric [8, 6] casting doubt on the validity of the results.”
Evaluating defect prediction approaches: a benchmark and an extensive comparison (2011, background): “Previous studies that advocated the absence of stability in the domains of effort estimation (Foss et al. 2003; Myrtveit et al. 2005) and defect prediction (Menzies et al.”