How, and why, process metrics are better

By Foyzur Rahman, Premkumar T. Devanbu. International Conference on Software Engineering, 2013

355 citations recorded by the source corpus.

Defect prediction techniques could potentially help us to focus quality-assurance efforts on the most defect-prone files. Modern statistical tools make it very easy to quickly build and deploy prediction models. Software metrics are at the heart of prediction models; understanding how and especially why different types of metrics are effective is very important for successful model deployment. In this paper we analyze the applicability and efficacy of process and code metrics from several different perspectives. We build many prediction models across 85 releases of 12 large open source projects to address the performance, stability, portability and stasis of different sets of metrics. Our results suggest that code metrics, despite widespread use in the defect prediction literature, are generally less useful than process metrics for prediction. Second, we find that code metrics have high stasis; they don't change very much from release to release. This leads to stagnation in the prediction models, leading to the same files being repeatedly predicted as defective; unfortunately, these recurringly defective files turn out to be comparatively less defect-dense.

Subfields: Computer Networks and Communications, Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/ef8ab10e584500052ab909b649c5e50f12c09994

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Software

Citations in the corpus

Deep Learning for Just-in-Time Defect Prediction (2015, methodology): “We use cost effectiveness and F1-score which are also used by past software engineering studies to evaluate the effectiveness of various prediction techniques [7], [12], [16], [33], [34], [17], [14], [13], [35].”
The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models (2018, background): “Recent studies [92], [93], [94], [95] recommend the consideration of developer effort when evaluating the performance of defect prediction models.”
An Empirical Comparison of Model Validation Techniques for Defect Prediction Models (2017, result): “Such unrealistic and unstable performance estimates could lead to incorrect model selection in practice and inaccurate conclusions for defect prediction studies [70], [105].”
Automated Parameter Optimization of Classification Techniques for Defect Prediction Models (2016, background): “, precision and recall) are problematic because they: (1) depend on an arbitrarily-selected threshold [32, 47] and (2) are sensitive to imbalanced data [18].”
The Impact of Automated Parameter Optimization on Defect Prediction Models (2018, background): “because they: (1) depend on an arbitrarily-selected threshold [1], [5], [80], [113], [124], [136] and (2) are sensitive to imbalanced data [24], [50], [84], [92], [132], [136].”
Learning to rank relevant files for bug reports using domain knowledge (2014, background): “It has been previously observed that software process metrics (e.g., change history) are more important than code metrics (e.g., size of codes) in detecting defects [39].”
Heterogeneous Defect Prediction (2015, methodology): “, McCabe’s cyclomatic, lines of code, and the number of developers modifying a file, follow this defect-proneness tendency [1], [2], [6], [15], [16].”
The impact of code review coverage and code review participation on software quality: a case study of the qt, VTK, and ITK projects (2014, neutral): “Rahman and Devanbu built defect prediction models to compare the impact of product and process metrics [33].”
On the "Naturalness" of Buggy Code (2015, methodology): “DP is implemented using two classifiers: logistic regression (LR) [43, 42] and Random Forest (RF), where the response is a binary variable indicating whether a file is buggy or not.”
An empirical study of the impact of modern code review practices on software quality (2015, background): “Rahman and Devanbu (2013) built defect prediction models to compare the impact of product and process metrics.”