Comparing Software Prediction Techniques Using Simulation

By Martin J. Shepperd, Gada F. Kadoda. IEEE Trans. Software Eng., 2001

305 citations recorded by the source corpus.

The need for accurate software prediction systems increases as software becomes much larger and more complex. We believe that the underlying characteristics: size, number of features, type of distribution, etc., of the data set influence the choice of the prediction system to be used. For this reason, we would like to control the characteristics of such data sets in order to systematically explore the relationship between accuracy, choice of prediction system, and data set characteristic. It would also be useful to have a large validation data set. Our solution is to simulate data allowing both control and the possibility of large (1000) validation cases. The authors compare four prediction techniques: regression, rule induction, nearest neighbor (a form of case-based reasoning), and neural nets. The results suggest that there are significant differences depending upon the characteristics of the data set. Consequently, researchers should consider prediction context when evaluating competing prediction systems. We observed that the more "messy" the data and the more complex the relationship with the dependent variable, the more variability in the results. In the more complex cases, we observed significantly different results depending upon the particular training set that has been sampled from the underlying data set. However, our most important result is that it is more fruitful to ask which is the best prediction system in a particular context rather than which is the "best" prediction system.

Subfields: Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/f5b701d5cc2000d22085d4a088543861342dc329

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

Citations in the corpus

Predicting defect-prone software modules using support vector machines (2008, methodology): “…(Schneidewind, 1992), optimized set reduction (Briand et al., 1993), genetic algorithms (Azar et al., 2002), classification trees (Selby and Porter, 1988), case‐based reasoning (Emam et al., 2001; Mair et al., 2000; Shepperd and Kadoda, 2001), and Dempster‐Shafer Belief Networks (Guo et al., 2003).”
Searching for rules to detect defective modules: A subgroup discovery approach (2012, background): “Also in this respect but in the domain of cost estimation, Shepperd and Kadoda [52] analyse the influence of different data characteristics (dataset size; number, type and independence of features; and type of distribution, in.”
An Empirical Comparison of Model Validation Techniques for Defect Prediction Models (2017, methodology): “Index Terms—Defect prediction models, model validation techniques, bootstrap validation, cross validation, holdout validation Ç”
Researcher Bias: The Use of Machine Learning in Software Defect Prediction (2014, background): “strated elsewhere through simulation studies [37].”
Software defect prediction using Bayesian networks (2012, methodology): “…defect prediction literature, there are many defect prediction algorithms studied like regression (Suffian and Abdullah 2010; Ekanayake et al. 2009; Shepperd and Kadoda 2001), rule induction (Shepperd and Kadoda 2001), decision tree approaches like C4.5 (Song et al. 2006), case-based reasoning…”
Systematic literature review of machine learning based software development effort estimation models (2012, unknown): “As pointed out in [63], choosing the best estimation method ‘‘in a particular context’’ is more fruitful than choosing the ‘‘best’’ estimation method.”
Reliability and validity in comparative studies of software prediction models (2005, methodology): “The latter include CART models [3], [5], [20], [35], OSR—Optimized Set Reduction, a subtype of CART [1], [2], [19], EBA models [17], [29], [33], [38], [41], [44], and, finally, ANN models [32], [35] [44].”