The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models

By Chakkrit Kla Tantithamthavorn, Ahmed E. Hassan, Ken-ichi Matsumoto. IEEE Transactions on Software Engineering, 2018

295 citations recorded by the source corpus.

Defect models that are trained on class imbalanced datasets (i.e., the proportion of defective and clean modules is not equally represented) are highly susceptible to produce inaccurate prediction models. Prior research compares the impact of class rebalancing techniques on the performance of defect models but arrives at contradictory conclusions due to the use of different choice of datasets, classification techniques, and performance measures. Such contradictory conclusions make it hard to derive practical guidelines for whether class rebalancing techniques should be applied in the context of defect models. In this paper, we investigate the impact of class rebalancing techniques on the performance measures and interpretation of defect models. We also investigate the experimental settings in which class rebalancing techniques are beneficial for defect models. Through a case study of 101 datasets that span across proprietary and open-source systems, we conclude that the impact of class rebalancing techniques on the performance of defect prediction models depends on the used performance measure and the used classification techniques. We observe that the optimized SMOTE technique and the under-sampling technique are beneficial when quality assurance teams wish to increase AUC and Recall, respectively, but they should be avoided when deriving knowledge and understandings from defect models.

Subfields: Artificial Intelligence, Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/e5482c28b119b6a18931346796f6fe13bae0fd47

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Software

Citations in the corpus

The Impact of Automated Parameter Optimization on Defect Prediction Models (2018, background): “because they: (1) depend on an arbitrarily-selected threshold [1], [5], [80], [113], [124], [136] and (2) are sensitive to imbalanced data [24], [50], [84], [92], [132], [136].”