Reliability and validity in comparative studies of software prediction models

By Ingunn Myrtveit, Erik Stensrud, Martin J. Shepperd. IEEE Transactions on Software Engineering, 2005

299 citations recorded by the source corpus.

Empirical studies on software prediction models do not converge with respect to the question "which prediction model is best?" The reason for this lack of convergence is poorly understood. In this simulation study, we have examined a frequently used research procedure comprising three main ingredients: a single data sample, an accuracy indicator, and cross validation. Typically, these empirical studies compare a machine learning model with a regression model. In our study, we use simulation and compare a machine learning and a regression model. The results suggest that it is the research procedure itself that is unreliable. This lack of reliability may strongly contribute to the lack of convergence. Our findings thus cast some doubt on the conclusions of any study of competing software prediction models that used this research procedure as a basis of model comparison. Thus, we need to develop more reliable research procedures before we can have confidence in the conclusions of comparative studies of software prediction models.

Subfields: Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/43288cf3876d73575b7397ee9ab2853cd4ff810a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

Citations in the corpus

A Systematic Literature Review on Fault Prediction Performance in Software Engineering (2012, unknown): “[25] for “more reliable research procedures before we can have confidence in the conclusions of comparative”
The Impact of Automated Parameter Optimization on Defect Prediction Models (2018, background): “[101] show that high variance in performance estimates from classification techniques is a critical problem in comparative studies of prediction models.”
Evaluating prediction systems in software project estimation (2012, unknown): “This was pointed out more than ten years ago by Kitchenham et al. [20] and subsequently by Foss et al. [9] and Myrtveit et al. [32].”
Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings (2008, methodology): “Furthermore, different accuracy indicators are used across studies, possibly leading to contradictory results [49], especially if these are based on the number of misclassified fp and nfp modules.”
Searching for rules to detect defective modules: A subgroup discovery approach (2012, background): “However, there are still large discrepancies regarding the assessment of the goodness of the different techniques and the reasons for such discrepancies [44,60,39].”
Systematic literature review of machine learning based software development effort estimation models (2012, methodology): “Besides SDEE models, many other prediction models in software engineering also suffer from this issue [68].”
Information needs for software development analytics (2012, background): “As such, these findings may not generalize [42], [59].”
Researcher Bias: The Use of Machine Learning in Software Defect Prediction (2014, unknown): “F”
Software defect prediction using Bayesian networks (2012, background): “According to Myrtveit et al. (2005), “we need to develop more reliable research procedures before we can have confidence in the conclusion of comparative studies of software prediction models”.”
A better measure of relative prediction accuracy for model selection and model estimation (2015, background): “Myrtveit et al. (2005) were concerned about the validity and reliability of studies which aimed to answer this question.”
Evaluating defect prediction approaches: a benchmark and an extensive comparison (2011, background): “Previous studies that advocated the absence of stability in the domains of effort estimation (Foss et al. 2003; Myrtveit et al. 2005) and defect prediction (Menzies et al.”
Automated Parameter Optimization of Classification Techniques for Defect Prediction Models (2016, background): “[42] show that high variance in performance estimates from classification techniques is a critical problem in comparative studies of prediction models.”
An Empirical Comparison of Model Validation Techniques for Defect Prediction Models (2017, methodology): “Index Terms—Defect prediction models, model validation techniques, bootstrap validation, cross validation, holdout validation Ç”
A General Software Defect-Proneness Prediction Framework (2011, result): “To reiterate a comment made in an earlier paper by one of the authors (Martin Shepperd) and also quoted by Lessmann et al. [24] " we need to develop more reliable research procedures before we can have confidence in the conclusion of comparative studies of software prediction models " [25].”