A study of the behavior of several methods for balancing machine learning training data

By Gustavo E. A. P. A. Batista, Ronaldo Cristiano Prati, Maria Carolina Monard. 2004

4,104 citations recorded by the source corpus.

There are several aspects that might influence the performance achieved by existing learning systems. It has been reported that one of these aspects is related to class imbalance in which examples in training data belonging to one class heavily outnumber the examples in the other class. In this situation, which is found in real world data describing an infrequent but important event, the learning system may have difficulties to learn the concept related to the minority class. In this work we perform a broad experimental evaluation involving ten methods, three of them proposed by the authors, to deal with the class imbalance problem in thirteen UCI data sets. Our experiments provide evidence that class imbalance does not systematically hinder the performance of learning systems. In fact, the problem seems to be related to learning with too few minority class examples in the presence of other complicating factors, such as class overlapping. Two of our proposed methods deal with these conditions directly, allying a known over-sampling method with data cleaning methods in order to produce better-defined class clusters. Our comparative experiments show that, in general, over-sampling methods provide more accurate results than under-sampling methods considering the area under the ROC curve (AUC). This result seems to contradict results previously published in the literature. Two of our proposed methods, Smote + Tomek and Smote + ENN, presented very good results for data sets with a small number of positive examples. Moreover, Random over-sampling, a very simple over-sampling method, is very competitive to more complex over-sampling methods. Since the over-sampling methods provided very good performance results, we also measured the syntactic complexity of the decision trees induced from over-sampled data. Our results show that these trees are usually more complex then the ones induced from original data. Random over-sampling usually produced the smallest increase in the mean number of induced rules and Smote + ENN the smallest increase in the mean number of conditions per rule, when compared among the investigated over-sampling methods.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Imbalanced Continual Learning with Partitioning Reservoir Sampling (2020, methodology): “Data processing methods such as oversampling or undersampling [5,8,65] explicitly simulate the input balance, while cost-sensitive approaches [38,85,18] adjust the update via regularizing the objective.”
A systematic study of the class imbalance problem in convolutional neural networks (2017, background): “[52] showed that most often the best strategy is to combine both undersampling and oversampling.”
A Systematic Literature Review on Fault Prediction Performance in Software Engineering (2012, neutral): “fault prediction (see [12], [27], [S179], [28], [29]).”
The Secrets of Machine Learning: Ten Things You Wish You Had Known Earlier to be More Effective at Data Analysis (2019, methodology): “To adjust the model to be appropriate for a new population, we might choose a weighting scheme or subsample of the data to mimic the correct population [4].”
How Complex Is Your Classification Problem? (2018, background): “In the particular case of class balance measures, Batista et al. [2004] show that the harmful effects due to class imbalance are more pronounced when there is also a large overlap between the classes.”
Machine Learning for Android Malware Detection Using Permission and API Calls (2013, methodology): “Following the pervious studies in validating the algorithms for imbalanced data [15], we chose precision and detection rate (Recall), in addition to AUC and accuracy, to evaluate our experiments.”
Editorial: special issue on learning from imbalanced data sets (2004, background): “al [6] present a comparison (and combination) of various sampling strategies.”
Data Mining for Imbalanced Datasets: An Overview (2005, background): “Over and under-sampling methodologies have received significant attention to counter the effect of imbalanced data sets (Solberg and Solberg, 1996; Japkowicz , 2000a; Chawla et al., 2002; Weiss and Provost, 2003; Kubat and Matwin, 1997; Jo and Japkowicz, 2004; Batista et al., 2004; Phua and Alahakoo”
A novel ensemble method for classifying imbalanced data (2015, methodology): “After that, we compared our method with the conventional external imbalance data handling methods, including random under-sampling [11], random over-sampling [11], SMOTE [22], MetaCost [38], Bagging [27], Boosting [28], EasyEnsemble [19], SMOTEBoost [15], RUSBoost [16], UnderBagging [39] and our pre”
Enhancing the effectiveness and interpretability of decision tree and rule induction classifiers with evolutionary training set selection over imbalanced problems (2009, methodology): “Solutions at the data level [8,3,9]: This kind of solution consists of balancing the class distribution by over-sampling the minority class (positive instances) or under-sampling the majority class (negative instances).”
Prototype Selection for Nearest Neighbor Classification: Taxonomy and Empirical Study (2012, methodology): “of classic PS methods to increase the data balance [61], [62].”
Deep Neural Networks for Bot Detection (2018, background): “These two combinations have been found to give excellent results on imbalanced data [5].”
Borderline-SMOTE: A New Over-Sampling Method in Imbalanced Data Sets Learning (2005, background): “combined over-sampling and under-sampling methods to resolve the imbalanced problem [14].”
On the use of MapReduce for imbalanced big data using Random Forest (2014, methodology): “The simplest undersampling method, random undersampling (RUS) [6], randomly deletes majority class examples from the original dataset until the balance with the minority class is achieved.”
Handling imbalanced datasets: A review (2006, background): “Clever re-sampling and combination methods can do quite more than cost-sensitive learning as they can provide new information or eliminate redundant information for the learning algorithm, as shown by [5], [6], [24], [14], [2].”