A survey on addressing high-class imbalance in big data

By Joffrey L. Leevy, Taghi M. Khoshgoftaar, Richard A. Bauder, Naeem Seliya. Journal of Big Data, 2018

675 citations recorded by the source corpus.

In a majority–minority classification problem, class imbalance in the dataset(s) can dramatically skew the performance of classifiers, introducing a prediction bias for the majority class. Assuming the positive (minority) class is the group of interest and the given application domain dictates that a false negative is much costlier than a false positive, a negative (majority) class prediction bias could have adverse consequences. With big data, the mitigation of class imbalance poses an even greater challenge because of the varied and complex structure of the relatively much larger datasets. This paper provides a large survey of published studies within the last 8 years, focusing on high-class imbalance (i.e., a majority-to-minority class ratio between 100:1 and 10,000:1) in big data in order to assess the state-of-the-art in addressing adverse effects due to class imbalance. In this paper, two techniques are covered which include Data-Level (e.g., data sampling) and Algorithm-Level (e.g., cost-sensitive and hybrid/ensemble) Methods. Data sampling methods are popular in addressing class imbalance, with Random Over-Sampling methods generally showing better overall results. At the Algorithm-Level, there are some outstanding performers. Yet, in the published studies, there are inconsistent and conflicting results, coupled with a limited scope in evaluated techniques, indicating the need for more comprehensive, comparative studies.

Subfields: Accounting, Artificial Intelligence, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/d99e88d3c1821857ca6945470698351925f9737f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

A review of ensemble learning and data augmentation models for class imbalanced problems: combination, implementation and evaluation (2023, background): “We need to develop better benchmark datasets to evaluate class imbalance problems to suit emerging models and big data problems [314].”
Boosting methods for multi-class imbalanced data classification: an experimental review (2020, background): “According to study [16], a dataset consisting at least 100,000 instances can be considered as a big dataset.”
Imbalance Problems in Object Detection: A Review (2019, background): “Another set of relevant work includes the studies specifically for imbalance problems in machine learning [40], [41], [42], [43].”