A General Software Defect-Proneness Prediction Framework

By Qinbao Song, Zihan Jia, Martin J. Shepperd, Shi Ying, Jin Liu. IEEE Transactions on Software Engineering, 2011

381 citations recorded by the source corpus.

BACKGROUND - Predicting defect-prone software components is an economically important activity and so has received a good deal of attention. However, making sense of the many, and sometimes seemingly inconsistent, results is difficult. OBJECTIVE - We propose and evaluate a general framework for software defect prediction that supports 1) unbiased and 2) comprehensive comparison between competing prediction systems. METHOD - The framework is comprised of 1) scheme evaluation and 2) defect prediction components. The scheme evaluation analyzes the prediction performance of competing learning schemes for given historical data sets. The defect predictor builds models according to the evaluated learning scheme and predicts software defects with new data according to the constructed model. In order to demonstrate the performance of the proposed framework, we use both simulation and publicly available software defect data sets. RESULTS - The results show that we should choose different learning schemes for different data sets (i.e., no scheme dominates), that small details in conducting how evaluations are conducted can completely reverse findings, and last, that our proposed framework is more effective and less prone to bias than previous approaches. CONCLUSIONS - Failure to properly or fully evaluate a learning scheme can be misleading; however, these problems may be overcome by our proposed framework.

Subfields: Computer Networks and Communications, Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/0f2e63430b5f6ac38cb635845463728550ea0f18

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Software

Citations in the corpus

Evaluating prediction systems in software project estimation (2012, background): “Although this might seem rather arcane, a study by Song et al. [37] illustrates how important using an unbiased estimator is.”
An Empirical Comparison of Model Validation Techniques for Defect Prediction Models (2017, background): “…validation technique is the number of Events Per Variable [2], [6], [82], [99], i.e., the ratio of the number of occurrences of the least frequently occurring class of the dependent variable (i.e., the events) to the number of independent variables used to train the model (i.e., the variables).”
The Impact of Automated Parameter Optimization on Defect Prediction Models (2018, background): “However, others [38], [70], [128] show that the performance of defect prediction models often depends on the characteristics of the defect datasets that are used to train them.”
Using Class Imbalance Learning for Software Defect Prediction (2013, methodology): “For example, the log filter was shown to improve the performance of Naive Bayes significantly, but contributed very little to decision trees [37].”
Automated Parameter Optimization of Classification Techniques for Defect Prediction Models (2016, background): “0 xG BT re e AV N N et G BM R F G PL S PD A N N et PM R G AM Bo os t PC AN N et M AR S FD A Ad aB oo st SV M R ad ia l M LP W ei gh tD ec ay M LP R BF N B R ip pe r LM T C AR T SV M Li ne ar J4 8 KN N Lo gi tB oo st L ik e lih o o d ●Optimized Classifier Default Classifier Figure 7: The likelihood…”
A Systematic Literature Review on Fault Prediction Performance in Software Engineering (2012, neutral): “This is an important finding as it suggests that a relatively high number of papers reporting fault prediction are not really doing any prediction (this finding is also reported by [6]).”
Data Quality: Some Comments on the NASA Software Defect Datasets (2013, background): “Lessman, Baesens, Mues and Pietsch [12] 2008 MDP × × × Liu, Khoshgoftaar and Seliya [15] 2010 MDP X X × Song, Jia, Shepperd, Ying and Liu [19] 2011 MDP X × ×”
Software defect prediction using ensemble learning on selected features (2015, methodology): “Decision trees [10,11], Bayesian methods [12], and ANNs [14] have paved the way for machine learning-based methods in the field of defect classification.”
A systematic review of machine learning techniques for software fault prediction (2015, background): “Hence, total number of 122 studies [1, 2, 20-139] were identified for further processing and analysis.”
An empirical study on software defect prediction with a simplified metric set (2014, methodology): “f independent variables in most data sets, it is useful to apply a “log-ﬁlter” to all numeric values v with log(v) (to avoid numeric errors with log(0), all numbers log(v) are replaced with log(v+1)) [38, 48]. In addition, there are many other commonly-used methods in machine learning literature, su”