Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach

By Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, Lizhen Qu. 2016

1,704 citations recorded by the source corpus.

We present a theoretically grounded approach to train deep neural networks, including recurrent networks, subject to class-dependent label noise. We propose two procedures for loss correction that are agnostic to both application domain and network architecture. They simply amount to at most a matrix inversion and multiplication, provided that we know the probability of each class being corrupted into another. We further show how one can estimate these probabilities, adapting a recent technique for noise estimation to the multi-class setting, and thus providing an end-to-end framework. Extensive experiments on MNIST, IMDB, CIFAR-10, CIFAR-100 and a large scale dataset of clothing images employing a diversity of architectures — stacking dense, convolutional, pooling, dropout, batch normalization, word embedding, LSTM and residual layers — demonstrate the noise robustness of our proposals. Incidentally, we also prove that, when ReLU is the only non-linearity, the loss curvature is immune to class-dependent label noise.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

On Robustness of Neural Architecture Search Under Label Noise (2020, background): “Great progress has been made in research on the robustness of learning algorithms under corrupted labels (Arpit et al., 2017; Chang et al., 2017; Ghosh et al., 2017; Patrini et al., 2017; Zhang et al., 2017, 2018; Jiang et al., 2018; Ren et al., 2018; Zhang and Sabuncu, 2018).”
Using Pre-Training Can Improve Model Robustness and Uncertainty (2019, methodology): “The Forward method of Patrini et al. (2017) uses a two-stage training procedure.”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, methodology): “We use the similar classes chosen in [63] to contaminate {20%, 40%} samples of the dataset with similar class pairs.”
Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty (2019, background): “Another robustness problem is learning in the presence of corrupted labels [Nettleton et al., 2010, Patrini et al., 2017].”
Are Labels Required for Improving Adversarial Robustness? (2019, background): “7 [36] G.”
Co-teaching: Robust training of deep neural networks with extremely noisy labels (2018, methodology): “[31] leveraged a two-step solution to estimating the noise transition matrix heuristically.”
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels (2018, methodology): “Forward T and T̂ represent forward correction with the true and estimated confusion matrices, respectively [31].”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, methodology): “isy Data Learning with noisy data can be divided into two approaches: (a) formulating explicit or implicit noise models to characterize the distribution of noisy and true labels using neural networks [14,21,33,60,56,73], graphical models [96,35], etc. and (b) using correction methods. E.g., relabeli”
Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise (2018, methodology): “[16] also obtains Ĉ by training a classifier on the noisy labels, and using the resulting softmax probabilities.”
Unsupervised label noise modeling and loss correction (2019, methodology): “Loss correction approaches (Reed et al., 2015; Jiang et al., 2018b; Patrini et al., 2017; Zhang et al., 2018) modify either the loss directly, or the probabilities used to compute it, to compensate for the incorrect guidance provided by the noisy samples.”
Model-Agnostic Approaches To Handling Noisy Labels When Training Sound Event Classifiers (2019, background): “, the probability of each true label being flipped into another [12, 13].”
Self-Training: A Survey (2022, positive): “, 2019] or estimating mislabeling errors by modeling the transition probability between noisy and true labels [Krithara et al., 2008; Patrini et al., 2017].”
L_DMI: An Information-theoretic Noise-robust Loss Function (2019, neutral): “We call our method DMI and compare it with: CE (the cross entropy loss), FW (the forward loss [29]), GCE (the generalized cross entropy loss [48]), LCCN (the latent class-conditional noise model [44]).”
Objective Robustness in Deep Reinforcement Learning (2021, background): “Approaches that address OOD robustness include learning causally invariant representations [5; 22] and, in the case of label noise, modifying the training objective [30].”
SELFIE: Refurbishing Unclean Samples for Robust Deep Learning (2019, methodology): “F-correction (Patrini et al., 2017) pre-trained a normal network to estimate the probability of each class being corrupted into another, in order to re-weight the forward or backward losses resulting from noisy labels.”
Improving Generalization by Controlling Label-Noise Information in Neural Network Weights (2020, methodology): “…Zhang & Sabuncu, 2018; Xu et al., 2019), loss correction techniques (Sukhbaatar et al., 2014; Tong Xiao et al., 2015; Goldberger & Ben-Reuven, 2017; Patrini et al., 2017), re-weighting samples (Jiang et al., 2017; Ren et al., 2018), detecting incorrect samples and re- labeling them (Reed et al.,…”
Fidelity-Weighted Learning (2017, background): “osed to remove or correct mislabeled samples (Brodley & Friedl, 1999). There are some studies showing that weak or noisy labels can be leveraged by modifying the loss function (Reed et al., 2015; Patrini et al., 2017; 2016; Vahdat, 2017) or changing the update rule to avoid imperfections of the ”