Evaluating prediction systems in software project estimation

By Martin J. Shepperd, Stephen G. MacDonell. Information and Software Technology, 2012

409 citations recorded by the source corpus.

Context: Software engineering has a problem in that when we empirically evaluate competing prediction systems we obtain conflicting results. Objective: To reduce the inconsistency amongst validation study results and provide a more formal foundation to interpret results with a particular focus on continuous prediction systems. Method: A new framework is proposed for evaluating competing prediction systems based upon (1) an unbiased statistic, Standardised Accuracy, (2) testing the result likelihood relative to the baseline technique of random 'predictions', that is guessing, and (3) calculation of effect sizes. Results: Previously published empirical evaluations of prediction systems are re-examined and the original conclusions shown to be unsafe. Additionally, even the strongest results are shown to have no more than a medium effect size relative to random guessing. Conclusions: Biased accuracy statistics such as MMRE are deprecated. By contrast this new empirical validation framework leads to meaningful results. Such steps will assist in performing future meta-analyses and in providing more robust and usable recommendations to practitioners.

Subfields: Computer Networks and Communications, Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/84ab8506d11673b1b69f6cb0ae36f093e6ca6777

Citations in the corpus

• A better measure of relative prediction accuracy for model selection and model estimation (2015, neutral): “example from Shepperd and MacDonell (2012): one project is predicted to cost 10 but actually costs 100; conversely, another project is predicted to cost 100 but ends up costing 10.”