Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings

By Stefan Lessmann, Bart Baesens, C. Mues, Swantje Pietsch. IEEE Transactions on Software Engineering, 2008

1,224 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/067c2a9ba5782065cc15722b3bdb7a168858982f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Software

Citations in the corpus

An exploratory study of the pull-based software development model (2014, methodology): “are known to perform well in large datasets [21] and have been used in previous work involving prediction models [13].”
Researcher Bias: The Use of Machine Learning in Software Defect Prediction (2014, neutral): “F”
Software fault prediction metrics: A systematic literature review (2013, methodology): “The performance difference be-tween modeling techniques appears to be moderate [38,30,60] and the choice of a modeling technique seems to have lesser impact on classiﬁcation accuracy of a model than the choice of a metrics set [60].”
Comparing and experimenting machine learning techniques for code smell detection (2015, methodology): “…that Random Forest tends to work quite well in various circumstances, see www.kaggle.com On a more general level, our results and the ones by (Lessmann et al, 2008) suggest that future comparison of machine learning on software engineering data should not be assessed on algorithmic…”
Easy over hard: a case study on deep learning (2017, background): “For example, two SE papers that use tuning [43, 64], apply a simple grid search to explore the potential parameter space for optimal tunings (such grid searchers run one for-loop for each parameter being optimized).”
Evaluating Complexity, Code Churn, and Developer Activity Metrics as Indicators of Software Vulnerabilities (2011, neutral): “[28] also reported that no significant difference in prediction performance was found between the 17 classification techniques they investigated.”
How, and why, process metrics are better (2013, background): “[12] decry the use of performance measures that require an arbitrary threshold.”
Evaluating defect prediction approaches: a benchmark and an extensive comparison (2011, methodology): “The existence of such data sets allows for systematic comparison, as the one by Menzies et al. (2010), or Lessmann et al. (2008) , where different prediction algorithms (Regression Models, Support Vector Machines, Random Forests, etc.), were trained on the same data and compared.”
A General Software Defect-Proneness Prediction Framework (2011, methodology): “Examples include [13], [26], [27], [28], [20], [23], and [24].”
A systematic review of machine learning techniques for software fault prediction (2015, methodology): “Menzies et Lessmann et al. [42 for construction is presented in F-measure It is the harmonic mean of precision and sensitivity.”
New insights into churn prediction in the telecommunication sector: A profit driven data mining approach (2012, methodology): “The applied techniques are evaluated using holdout evaluation on a single random split up of each data set into 2/3 training set and 1/3 test set, as commonly applied in large-scale benchmarking studies (e.g. Baesens et al., 2003b; Lessmann et al., 2008).”
The Impact of Automated Parameter Optimization on Defect Prediction Models (2018, methodology): “because they: (1) depend on an arbitrarily-selected threshold [1], [5], [80], [113], [124], [136] and (2) are sensitive to imbalanced data [24], [50], [84], [92], [132], [136].”
Explaining Data-Driven Document Classifications (2014, methodology): “…studies have revealed that when it comes to predictive accuracy, non-linear methods often outperform traditional statistical methods such as multiple regression, logistic regression, naive Bayesian and linear discriminant analysis (see, e.g. Baesens et al. (2003), Lessmann et al. (2008)).”
Heterogeneous Defect Prediction (2015, background): “To address the limitation, we propose heterogeneous defect prediction (HDP) to predict defects across projects with heterogeneous metric sets.”
On the relative value of cross-company and within-company data for defect prediction (2009, methodology): “investigate the statistical difference of the results between 19 learners, including naive Bayes, on the NASA projects [17].”
A large-scale empirical study of just-in-time quality assurance (2013, methodology): “To get an overall idea of the performance across thresholds, we additionally use the area under the curve (AUC) of ROC (the receiver operating characteristics) [28].”
Searching for rules to detect defective modules: A subgroup discovery approach (2012, methodology): “[33] compare 22 classifiers grouped into statistical, nearest neighbour methods, neural networks, support vector machine, decision trees and ensemble methods over ten datasets from the NASA repository.”
Automated Parameter Optimization of Classification Techniques for Defect Prediction Models (2016, methodology): “, precision and recall) are problematic because they: (1) depend on an arbitrarily-selected threshold [32, 47] and (2) are sensitive to imbalanced data [18].”