A systematic study of the class imbalance problem in convolutional neural networks

By Mateusz Buda, Atsuto Maki, Maciej A. Mazurowski. 2017

3,026 citations recorded by the source corpus.

In this study, we systematically investigate the impact of class imbalance on classification performance of convolutional neural networks (CNNs) and compare frequently used methods to address the issue. Class imbalance is a common problem that has been comprehensively studied in classical machine learning, yet very limited systematic research is available in the context of deep learning. In our study, we use three benchmark datasets of increasing complexity, MNIST, CIFAR-10 and ImageNet, to investigate the effects of imbalance on classification and perform an extensive comparison of several methods to address the issue: oversampling, undersampling, two-phase training, and thresholding that compensates for prior class probabilities. Our main evaluation metric is area under the receiver operating characteristic curve (ROC AUC) adjusted to multi-class tasks since overall accuracy metric is associated with notable difficulties in the context of imbalanced data. Based on results from our experiments we conclude that (i) the effect of class imbalance on classification performance is detrimental; (ii) the method of addressing class imbalance that emerged as dominant in almost all analyzed scenarios was oversampling; (iii) oversampling should be applied to the level that completely eliminates the imbalance, whereas the optimal undersampling ratio depends on the extent of imbalance; (iv) as opposed to some classical machine learning models, oversampling does not cause overfitting of CNNs; (v) thresholding should be applied to compensate for prior class probabilities when overall number of properly classified cases is of interest.

Subfields: Artificial Intelligence, Civil and Structural Engineering, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Percy Liang
  2. Mohit Bansal
  3. Jianfeng Gao
  4. Quoc V. Le
  5. Yoshua Bengio
  6. Sergey Levine
  7. Dacheng Tao
  8. Trevor Darrell

In Civil and Structural Engineering

  1. Adrian Daniel Popescu
  2. Eden Belouadah
  3. Percy Liang
  4. Shiori Sagawa
  5. Pang Wei Koh
  6. Zhi Xu
  7. Yuzhe Yang
  8. Damien Teney

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Xiaogang Wang
  4. Ming-Hsuan Yang
  5. Trevor Darrell
  6. Bernt Schiele
  7. Jitendra Malik
  8. Andrew Zisserman

In Electrical and Electronic Engineering

  1. Chelsea Finn
  2. Abhishek Gupta
  3. Adrian Daniel Popescu
  4. Eden Belouadah
  5. Sergey Levine
  6. Anton van den Hengel
  7. Percy Liang
  8. Shiori Sagawa

In Optimization & Training

  1. Percy Liang
  2. Shiori Sagawa
  3. Pang Wei Koh
  4. Bingyi Kang
  5. Jiashi Feng
  6. Chelsea Finn
  7. Abhishek Gupta
  8. Yu Li

Citations in the corpus