The impact of class imbalance in classification performance metrics based on the binary confusion matrix

By Amalia Luque, A. Carrasco, Alejandro Martín, A. D. L. Heras. Pattern Recognition, 2019

1,018 citations recorded by the source corpus.

Abstract A major issue in the classification of class imbalanced datasets involves the determination of the most suitable performance metrics to be used. In previous work using several examples, it has been shown that imbalance can exert a major impact on the value and meaning of accuracy and on certain other well-known performance metrics. In this paper, our approach goes beyond simply studying case studies and develops a systematic analysis of this impact by simulating the results obtained using binary classifiers. A set of functions and numerical indicators are attained which enables the comparison of the behaviour of several performance metrics based on the binary confusion matrix when they are faced with imbalanced datasets. Throughout the paper, a new way to measure the imbalance is defined which surpasses the Imbalance Ratio used in previous studies. From the simulation results, several clusters of performance metrics have been identified that involve the use of Geometric Mean or Bookmaker Informedness as the best null-biased metrics if their focus on classification successes (dismissing the errors) presents no limitation for the specific application where they are used. However, if classification errors must also be considered, then the Matthews Correlation Coefficient arises as the best choice. Finally, a set of null-biased multi-perspective Class Balance Metrics is proposed which extends the concept of Class Balance Accuracy to other performance metrics.

Subfields: Accounting, Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/7491760fdef8f8703c18428c08547530dc839bc5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qing He
2. Rohitash Chandra
3. Zidi Qin
4. Hao Yang
5. Jinghua Feng
6. Yang Liu
7. Azal Ahmad Khan
8. Omkar Chaudhari

In Information Systems

1. Qing He
2. Rohitash Chandra
3. Azal Ahmad Khan
4. Omkar Chaudhari
5. Xiang Ao

Citations in the corpus

• Credit Card Fraud Detection using Machine Learning Algorithms (2019, neutral): “In this case, measures based on the confusion matrix are used (Luque et al., 2019).”
• A review of ensemble learning and data augmentation models for class imbalanced problems: combination, implementation and evaluation (2023, background): “The classification accuracy fails to capture model performance in CI problems such as the minority classes [303, 304], since it does not account for the di ff erence in the size of the classes.”
• Pick and Choose: A GNN-based Imbalanced Learning Approach for Fraud Detection (2021, background): “For class imbalance classification, the evaluation metrics should have no bias to any class[27].”