Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss

By Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Aréchiga, Tengyu Ma. 2019

2,156 citations recorded by the source corpus.

Deep learning algorithms can fare poorly when the training dataset suffers from heavy class-imbalance but the testing criterion requires good generalization on less frequent classes. We design two novel methods to improve performance in such scenarios. First, we propose a theoretically-principled label-distribution-aware margin (LDAM) loss motivated by minimizing a margin-based generalization bound. This loss replaces the standard cross-entropy objective during training and can be applied with prior strategies for training with class-imbalance such as re-weighting or re-sampling. Second, we propose a simple, yet effective, training schedule that defers re-weighting until after the initial stage, allowing the model to learn an initial representation while avoiding some of the complications associated with re-weighting or re-sampling. We test our methods on several benchmark vision tasks including the real-world imbalanced dataset iNaturalist 2018. Our experiments show that either of these methods alone can already improve over existing techniques and their combination achieves even better performance gains.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation (2020, background): “Modern approaches for addressing long-tail data when training deep networks can be mainly divided into two groups: data re-sampling [41, 21, 62] and loss re-weighting [30, 8, 3, 54, 37, 46].”
Long-Tailed Class Incremental Learning (2022, background): “The long-tailed distribution follows an exponential decay in sample sizes across classes as described in [6].”
Supervised Contrastive Learning (2020, background): “A number of papers have studied other drawbacks of the cross-entropy loss, such as sensitivity to noisy labels [63, 46], presence of adversarial examples [10, 36], and poor margins [2].”
Decoupling Representation and Classifier for Long-Tailed Recognition (2019, methodology): “Various re-sampling strategies (Chawla et al., 2002; Shen et al., 2016; Cao et al., 2019), loss reweighting and margin regularization over few-shot classes are thus proposed.”
Does learning require memorization? a short tale about a long tail (2019, background): “such modeling has practical benefits can be found in [15].”
Learning Adaptive Classifiers Synthesis for Generalized Few-Shot Learning (2019, background): “Directly training a system over all classes is prone to over-fit and can be biased towards the data-rich categories (Cui et al. 2019; Cao et al. 2019; Kang et al. 2020; Ye et al. 2020; Zhou et al. 2020).”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “, 2019); label-distribution-aware margin losses (Cao et al., 2019); adaptive Lipschitz regularization (Cao et al.”
Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning (2020, background): “Cross entropy loss leads to poor generalization performance due to poor margins (Liu et al., 2016; Cao et al., 2019), and it lacks robustness to noisy labels (Zhang & Sabuncu, 2018; Sukhbaatar et al., 2015) or adversarial examples (Elsayed et al., 2018; Nar et al., 2019).”
Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization (2019, methodology): “This gives the group-adjusted DRO estimator The scaling with 1 / √ n g reﬂects how smaller groups are more prone to overﬁtting than larger groups, and is inspired by the general size dependence of model-complexity-based generalization bounds (see, e.g., Cao et al. (2019)).”
An Investigation of Why Overparameterization Exacerbates Spurious Correlations (2020, background): “Similarly Cao et al. (2019) show that data-dependent regularization can improve error on rare labels.”
BENDR: Using Transformers and a Contrastive Self-Supervised Learning Task to Learn From Massive Amounts of EEG Data (2021, background): “This would no doubt be imbalanced; the labels would be distributed in a long-tailed or Zipfian distribution that would likely require well thought-out adjustment (Cao et al., 2019; Tang et al., 2020).”
Zero-Shot Detection via Vision and Language Knowledge Distillation (2021, methodology): “Specialized methods are proposed to address the data imbalance issue, including two-stage training [24, 26], loss re-weighting [19, 6, 5, 40, 26], and data re-sampling [30, 15, 42].”
Distributionally Robust Neural Networks (2020, methodology): “, Cao et al. (2019)).”
Long-tail learning via logit adjustment (2020, methodology): “, 2020) Adaptive margin Softmax with rare +ve upweighting × (Cao et al., 2019) Equalised margin Softmax with rare -ve downweighting × (Tan et al.”
Disentangling Label Distribution for Long-tailed Visual Recognition (2020, methodology): “Evaluation results on CIFAR-100-LT, Places-LT, ImageNet-LT, and iNaturalist 2018 are shown in Table 2, 3, 4, and 5, respectively.”
Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets (2020, background): “It shares a similar idea with [3] that a larger margin is needed for rare classes.”
Overcoming Classifier Imbalance for Long-Tail Object Detection With Balanced Group Softmax (2020, methodology): “One may consider using solutions to long-tail classiﬁcation to overcome such an issue, including re-sampling training instances to balance the distribution [16, 8, 34, 27] and re-weighting classiﬁcation loss at category level [6, 2, 19] or instance level [24, 35].”
Delving into Deep Imbalanced Regression (2021, neutral): “This phenomenon poses great challenges for deep recognition models, and has motivated many prior techniques for addressing data imbalance (Cao et al., 2019; Cui et al., 2019; Huang et al., 2019; Liu et al., 2019; Tang et al., 2020).”
Rethinking Data Augmentation: Self-Supervision and Self-Distillation (2019, methodology): “For this experiment, we combine our SDA+SD method with two recent approaches, Class-Balanced (CB) loss (Cui et al., 2019) and LDAM (Cao et al., 2019), specialized for this problem.”
Self-supervised Label Augmentation via Input Transformations (2019, methodology): “When combining with other methods, we use publicly available codes and follow their experimental setups: MetaOptNet (Lee et al., 2019) for few-shot learning, LDAM (Cao et al., 2019) for imbalanced datasets, and FastAutoAugment (Lim et al., 2019) and CutMix (Yun et al., 2019) for advanced augmentatio”