Detecting representative data and generating synthetic samples to improve learning accuracy with imbalanced data sets

By Der-Chiang Li, Susan C. Hu, Liang-Sian Lin, Chun-Wu Yeh. 2017

32 citations recorded by the source corpus.

It is difficult for learning models to achieve high classification performances with imbalanced data sets, because with imbalanced data sets, when one of the classes is much larger than the others, most machine learning and data mining classifiers are overly influenced by the larger classes and ignore the smaller ones. As a result, the classification algorithms often have poor learning performances due to slow convergence in the smaller classes. To balance such data sets, this paper presents a strategy that involves reducing the sizes of the majority data and generating synthetic samples for the minority data. In the reducing operation, we use the box-and-whisker plot approach to exclude outliers and the Mega-Trend-Diffusion method to find representative data from the majority data. To generate the synthetic samples, we propose a counterintuitive hypothesis to find the distributed shape of the minority data, and then produce samples according to this distribution. Four real datasets were used to examine the performance of the proposed approach. We used paired t-tests to compare the Accuracy, G-mean, and F-measure scores of the proposed data pre-processing (PPDP) method merging in the D3C method (PPDP+D3C) with those of the one-sided selection (OSS), the well-known SMOTEBoost (SB) study, and the normal distribution-based oversampling (NDO) approach, and the proposed data pre-processing (PPDP) method. The results indicate that the classification performance of the proposed approach is better than that of above-mentioned methods.

Subfields: Artificial Intelligence, Statistics and Probability, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Ben Hutchinson
  2. Inioluwa Deborah Raji
  3. Timnit Gebru
  4. Margaret Mitchell
  5. Lucy Vasserman
  6. Andrew Zaldivar
  7. Parker Barnes
  8. Daniel S. Schiff

In Statistics and Probability

  1. Lucy Vasserman
  2. Ben Hutchinson
  3. Timnit Gebru
  4. Margaret Mitchell
  5. Inioluwa Deborah Raji

In Evaluation & Benchmarks

  1. Inioluwa Deborah Raji
  2. Margaret Mitchell
  3. Lucy Vasserman
  4. Ben Hutchinson
  5. Andrew Zaldivar
  6. Timnit Gebru
  7. Parker Barnes
  8. Jordan L. Boyd-Graber

Citations in the corpus