Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels

By Zhilu Zhang, Mert Rory Sabuncu. 2018

3,268 citations recorded by the source corpus.

Deep neural networks (DNNs) have achieved tremendous success in a variety of applications across many disciplines. Yet, their superior performance comes with the expensive cost of requiring correctly annotated large-scale datasets. Moreover, due to DNNs’ rich capacity, errors in training labels can hamper performance. To combat this problem, mean absolute error (MAE) has recently been proposed as a noise-robust alternative to the commonly-used categorical cross entropy (CCE) loss. However, as we show in this paper, MAE can perform poorly with DNNs and challenging datasets. Here, we present a theoretically grounded set of noise-robust loss functions that can be seen as a generalization of MAE and CCE. Proposed loss functions can be readily applied with any existing DNN architecture and algorithm, while yielding good performance in a wide range of noisy label scenarios. We report results from experiments conducted with CIFAR-10, CIFAR-100 and FASHION-MNIST datasets and synthetically generated noisy labels.

Subfields: Artificial Intelligence, Optimization & Training, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Safety, Robustness & Interpretability

Citations in the corpus

Using Pre-Training Can Improve Model Robustness and Uncertainty (2019, background): “Zhang & Sabuncu (2018) show that networks overﬁt to the incorrect labels when trained for too long (Figure 1).”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, methodology): “There have been multiple lines of work to learn amidst noisy labels such as loss regularization [84, 96, 31], data re-weighting [68, 72], label cleaning [64, 42, 61], and”
Supervised Contrastive Learning (2020, background): “A number of papers have studied other drawbacks of the cross-entropy loss, such as sensitivity to noisy labels [63, 46], presence of adversarial examples [10, 36], and poor margins [2].”
Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty (2019, background): “Fine-tuning for longer can eliminate this gap, but also leads to overfitting label noise [Zhang and Sabuncu, 2018].”
Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning (2020, background): “…cross-entropy loss, demonstrating that it leads to poor generalization performance due to poor margins (Liu et al., 2016; Cao et al., 2019), and lack of robustness to noisy labels (Zhang & Sabuncu, 2018; Sukhbaatar et al., 2015) or adversarial examples (Elsayed et al., 2018; Nar et al., 2019).”
Semantic Segmentation of Underwater Imagery: Dataset and Benchmark (2020, positive): “The end-to-end training is supervised by the standard cross-entropy loss [51], [3] which evaluates the discrepancy between the predicted and ground truth pixel labels.”
DeepSMOTE: Fusing Deep Learning and SMOTE for Imbalanced Data (2021, methodology): “More recently, multiple other loss functions were proposed, such as Log Bilinear Loss [75], Cross Entropy Loss [76], and Class-Balanced Loss [77].”
NLNL: Negative Learning for Noisy Labels (2019, methodology): “Some methods attempted to create noise-robust losses [1, 32, 23, 4, 3, 39].”
Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks (2019, background): “In addition to early stopping, several heuristics have been proposed to speciﬁcally deal with label noise [26,30,36,42,47,57].”
Are Anchor Points Really Indispensable in Label-Noise Learning? (2019, background): “For example, loss functions are modiﬁed to ensure risk consistency, e.g., [49, 17, 22, 29, 35, 26]; a noise adaptation layer is added to deep neural networks to design classiﬁer-consistent deep learning algorithms [9, 30, 38, 47].”
Model-Agnostic Approaches To Handling Noisy Labels When Training Sound Event Classifiers (2019, methodology): “Nevertheless, MAE has been reported to cause other difficulties in training that lead to performance drop [5, 14].”
Tilted Empirical Risk Minimization (2020, methodology): “Following the setup in [25], we explore classiﬁcation on CIFAR-10 [32] when a fraction of the training labels are corrupted with uniform noise—comparing TERM with ERM and several state-of-the-art approaches [32, 33, 50, 74].”
Symmetric Cross Entropy for Robust Learning With Noisy Labels (2019, methodology): “Whilst for asymmetric noisy labels, flipping labels only occurs within a specific set of classes [16, 29], for example, for MNIST, flipping 2 → 7, 3 → 8, 5 ↔ 6 and 7 → 1; for CIFAR-10, flipping TRUCK → AUTOMOBILE, BIRD → AIRPLANE, DEER → HORSE, CAT ↔ DOG; for CIFAR-100, the 100 classes are grouped i”
Learning from Rules Generalizing Labeled Exemplars (2020, methodology): “If we train a classiﬁer with many examples in H 1 ∪ H 2 wrongly labeled by rules, then even with a noise tolerant loss function like Zhang & Sabuncu (2018), the classiﬁer P θ ( y | x ) might be misled.”
Part-dependent Label Noise: Towards Instance-dependent Label Noise (2020, background): “Label noise models Currently, there are three typical label noise models, i.e., the random classiﬁ-cation noise (RCN) model [8, 44, 42], the class-conditional label noise (CCN) model [50, 67, 79], and the instance-dependent label noise (IDN) model [6, 9, 12].”
GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning (2020, methodology): “2018; Malach and Shalev-Shwartz 2017) or by using a clean held out validation set (Ren et al. 2018; Zhang and Sabuncu 2018).”
Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting (2019, methodology): “[17] Zhilu Zhang and Mert R Sabuncu.”