Robust Loss Functions under Label Noise for Deep Neural Networks

By Aritra Ghosh, Himanshu Kumar, P. Shanti Sastry. 2017

1,168 citations recorded by the source corpus.

In many applications of classifier learning, training data suffers from label noise. Deep networks are learned using huge training data where the problem of noisy labels is particularly relevant. The current techniques proposed for learning deep networks under label noise focus on modifying the network architecture and on algorithms for estimating true labels from noisy labels. An alternate approach would be to look for loss functions that are inherently noise-tolerant. For binary classification there exist theoretical results on loss functions that are robust to label noise. In this paper, we provide some sufficient conditions on a loss function so that risk minimization under that loss function would be inherently tolerant to label noise for multiclass classification problems. These results generalize the existing results on noise-tolerant loss functions for binary classification. We study some of the widely used loss functions in deep networks and show that the loss function based on mean absolute value of error is inherently robust to label noise. Thus standard back propagation is enough to learn the true classifier even under label noise. Through experiments, we illustrate the robustness of risk minimization with such loss functions for learning neural networks.

Subfields: Artificial Intelligence, Civil and Structural Engineering, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Civil and Structural Engineering

In Computational Theory and Mathematics

Citations in the corpus

On Robustness of Neural Architecture Search Under Label Noise (2020, methodology): “Recently, some robust loss functions are suggested for learning the weights of a network under label noise (Ghosh et al., 2017; Zhang and Sabuncu, 2018).”
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels (2018, background): “[9] who theoretically showed that mean absolute error (MAE) can be robust against noisy labels under certain assumptions.”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, methodology): “…the data during training (Reed et al. 2015), jointly optimizing the model’s parameters and estimating true labels (Tanaka et al. 2018), using noise-tolerant loss function (Ghosh, Kumar, and Sastry 2017; Van Rooyen, Menon, and Williamson 2015) or noise tolerant training algorithms (Li et al. 2019).”
NLNL: Negative Learning for Noisy Labels (2019, methodology): “82 MAE [3] 80.”
Robust Loss Functions for Learning Multi-class Classifiers (2018, background): “The theoretical result for symmetric label noise in [20] is as follows: Theorem 1: In a multi-class classiﬁcation problem, let loss function L satisfy Eq.”
A Noise-Robust Framework for Automatic Segmentation of COVID-19 Pneumonia Lesions From CT Images (2020, methodology): “…noisy annotations for training CNNs to segment COVID-19 pneumonia lesions, we propose a novel noise-robust Dice loss function, which is a combination and generalization of MAE loss [14] that is robust against noisy labels and Dice loss [18] that is insensitive to foreground-background imbalance.”
Orthogonal Projection Loss (2021, neutral): “Multiple works [13, 65] explore modiﬁcations to common objective functions aimed at building robustness to label noise.”
Dimensionality-Driven Learning with Noisy Labels (2018, methodology): “A different approach to tackle noisy labels is to utilize correction methods such as loss correction (Patrini et al., 2017; Ghosh et al., 2017), label correction (Reed et al.”
An Incremental Self-Labeling Strategy for Semi-Supervised Deep Learning Based on Generative Adversarial Networks (2020, background): “However, in most machine learning methods, especially in deep learning, the training samples with label noise are often destructive [16], [17].”
Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey (2019, background): “Robust Losses Non-convex loss functions [103]–[105], 0-1 loss surrogate [106], MAE [107], IMEA [108], Generalized cross-entropy [109], symmetric loss [110], unbiased estimator [111], modified cross-entropy for omission [112], information theoric loss [113], linear-odd losses [114], classification ca”
Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations (2021, background): “…the loss to encourage deep neural nets to fit on correct labels [Liu and Tao, 2015], propose robust loss functions [Natarajan et al., 2013, Ghosh et al., 2017, Zhang and Sabuncu, 2018, Amid et al., 2019, Wang et al., 2019, Liu and Guo, 2020], or introduce a robust regularizer [Liu et al.,…”
Normalized Loss Functions for Deep Learning with Noisy Labels (2020, background): “Following (Ghosh et al., 2017; Charoenphakdee et al., 2019), we know that if a loss function L satisﬁes L ( f ( x ) , j ) = C, ∀ x ∈ X , ∀ f , where C is some constant, then L is noise tolerant under mild assumptions.”
Early-Learning Regularization Prevents Memorization of Noisy Labels (2020, background): “These include Mean Absolute Error (MAE) [10], Generalized Cross Entropy [54], which can be interpreted as a generalization of MAE, Symmetric Cross Entropy [43], which adds a reverse cross-entropy term to the usual cross-entropy loss, and L DIM [46], which is based on information-theoretic…”
Model-Agnostic Approaches To Handling Noisy Labels When Training Sound Event Classifiers (2019, methodology): “In contrast, the mean absolute error equation (MAE) treats every instance equally (hence avoiding the weighting aspect), and it has been shown theoretically that MAE can be used as a loss function robust against label noise [22].”
Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification (2020, methodology): “Noise-robust methods designed robust loss functions against label noises, for instance, Mean Absolute Error (MAE) loss (Ghosh et al., 2017), Generalized Cross Entropy (GCE) loss (Zhang & Sabuncu, 2018) and Label Smoothing Regularization (LSR) (Szegedy et al., 2016).”
Learning to Learn From Noisy Labeled Data (2018, methodology): “A few other methods have also been proposed that use noisetolerant loss functions to achieve robust learning under label noise [3, 4, 28].”
Deep learning with noisy labels: exploring techniques and remedies in medical image analysis (2019, methodology): “Loss functions [80] - image and text classification lesion detection (experiments in Section V-A), pathology classification (experiments in Section V-B), segmentation ([81]; experiments in Section V-C) [82] - image classification [83] - image classification, object detection [84] - image classificat”
Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls (2022, background): “…proposed solutions into four categories: robust architectures (e.g., (Sukhbaatar et al., 2014; Xiao et al., 2015)), robust loss functions (e.g., (Ghosh et al., 2017; Liu and Guo, 2020)), robust regularization (e.g., (Jenni and Favaro, 2018; Goodfellow et al., 2014)) and sample selection…”