A systematic study of the class imbalance problem in convolutional neural networks
By Mateusz Buda, Atsuto Maki, Maciej A. Mazurowski. 2017
3,026 citations recorded by the source corpus.
In this study, we systematically investigate the impact of class imbalance on classification performance of convolutional neural networks (CNNs) and compare frequently used methods to address the issue. Class imbalance is a common problem that has been comprehensively studied in classical machine learning, yet very limited systematic research is available in the context of deep learning. In our study, we use three benchmark datasets of increasing complexity, MNIST, CIFAR-10 and ImageNet, to investigate the effects of imbalance on classification and perform an extensive comparison of several methods to address the issue: oversampling, undersampling, two-phase training, and thresholding that compensates for prior class probabilities. Our main evaluation metric is area under the receiver operating characteristic curve (ROC AUC) adjusted to multi-class tasks since overall accuracy metric is associated with notable difficulties in the context of imbalanced data. Based on results from our experiments we conclude that (i) the effect of class imbalance on classification performance is detrimental; (ii) the method of addressing class imbalance that emerged as dominant in almost all analyzed scenarios was oversampling; (iii) oversampling should be applied to the level that completely eliminates the imbalance, whereas the optimal undersampling ratio depends on the extent of imbalance; (iv) as opposed to some classical machine learning models, oversampling does not cause overfitting of CNNs; (v) thresholding should be applied to compensate for prior class probabilities when overall number of properly classified cases is of interest.
Subfields: Artificial Intelligence, Civil and Structural Engineering, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Mohit Bansal
- Jianfeng Gao
- Quoc V. Le
- Yoshua Bengio
- Sergey Levine
- Dacheng Tao
- Trevor Darrell
In Civil and Structural Engineering
- Adrian Daniel Popescu
- Eden Belouadah
- Percy Liang
- Shiori Sagawa
- Pang Wei Koh
- Zhi Xu
- Yuzhe Yang
- Damien Teney
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Ming-Hsuan Yang
- Trevor Darrell
- Bernt Schiele
- Jitendra Malik
- Andrew Zisserman
In Electrical and Electronic Engineering
- Chelsea Finn
- Abhishek Gupta
- Adrian Daniel Popescu
- Eden Belouadah
- Sergey Levine
- Anton van den Hengel
- Percy Liang
- Shiori Sagawa
In Optimization & Training
Citations in the corpus
- Adversarial Training for High-Stakes Reliability (2022, background): “Because our datasets had more non-injurious examples than injurious examples, we upsampled [46] snippets labeled injurious by up to 5 so that they were closer to the number of non-injurious examples.”
- Differential Privacy Has Disparate Impact on Model Accuracy (2019, background): “Other approaches to balancing accuracy across classes include oversampling [8], adversarial training [3] with a loss function that overweights the underrepresented group, cost-sensitive learning [9], and re-sampling [6].”
- Imbalanced Continual Learning with Partitioning Reservoir Sampling (2020, methodology): “Data processing methods such as oversampling or undersampling [5,8,65] explicitly simulate the input balance, while cost-sensitive approaches [38,85,18] adjust the update via regularizing the objective.”
- Maintaining Discrimination and Fairness in Class Incremental Learning (2019, background): “Though class imbalance is an old topic and has attracted a lot of attention [4, 15, 17], multi-class imbalance learning is still an open problem [35].”
- IL2M: Class Incremental Learning With Dual Memory (2019, methodology): “We will pursue work along the following lines: (1) test a constant complexity method such as IL2M for multitask IL to replicate real-life scenarios in which more diversified visual content is encountered, (2) enhance vanilla fine tuning by leveraging recent results which improve imbalanced learning ”
- Revisiting Distillation and Incremental Classifier Learning (2018, background): “[3] showed that threshold moving is effective in removing bias in ANNs.”
- A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020, methodology): “The well-known thresholding method [8] is used to reduce the classification bias between past and new classes.”
- FLEX: Unifying Evaluation for Few-Shot NLP (2021, background): “Class imbalance has also been observed to degrade performance [11, 47], yet prior few-shot NLP evaluations do not include this setting either.”
- CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning (2017, methodology): “We train the model using minibatches of size 16, where we oversample the minority (positive) class (Buda et al., 2017).”
- Class-Balanced Loss Based on Effective Number of Samples (2019, background): “CNNs trained on such data perform poorly for weakly represented classes [19, 15, 41, 4].”
- CVPR 2020 Continual Learning in Computer Vision Competition: Approaches, Results, Current Challenges and Future Directions (2020, methodology): “For NIC track, based on the assumption that the neural network estimates Bayesian a posteriori probilitities (Richard & Lippmann, 1991), the network outputs are divided by the prior probability for each class inspired by the trick that handles class imbalance (Buda et al., 2018).”
- Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization (2019, background): “For label shifts, prior work has observed overfitting on rare labels and proposed algorithms to mitigate it (Buda et al., 2018; Cui et al., 2019; Cao et al., 2019).”
- Augmentation for small object detection (2019, background): “Oversampling We address the issue of relatively fewer images containing small objects by oversampling those images during training [4].”
- Contrastive Learning based Hybrid Networks for Long-Tailed Image Classification (2021, methodology): “Typical methods include data re-sampling [1, 8, 35, 4, 28], loss re-weighting [26, 7, 30, 33], margin modification [3], and data augmentation [19, 6, 29].”
- Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm (2019, methodology): “Therefore, to account for this fact, we applied oversampling with data augmentation that was proved to help in training convolutional neural networks [23].”
- Balanced Meta-Softmax for Long-Tailed Visual Recognition (2020, background): “Specifically, re-sampling strategies [19, 5, 8, 10, 26, 2, 1] try to restore the true distributions from the imbalanced training data.”