Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise

By Dan Hendrycks, Mantas Mazeika, Duncan Wilson, Kevin Gimpel. 2018

616 citations recorded by the source corpus.

The growing importance of massive datasets with the advent of deep learning makes robustness to label noise a critical property for classifiers to have. Sources of label noise include automatic labeling for large datasets, non-expert labeling, and label corruption by data poisoning adversaries. In the latter case, corruptions may be arbitrarily bad, even so bad that a classifier predicts the wrong labels with high confidence. To protect against such sources of noise, we leverage the fact that a small set of clean labels is often easy to procure. We demonstrate that robustness to label noise up to severe strengths can be achieved by using a set of trusted data with clean labels, and propose a loss correction that utilizes trusted examples in a data-efficient manner to mitigate the effects of label noise on deep neural network classifiers. Across vision and natural language processing tasks, we experiment with various label noises at several strengths, and show that our method significantly outperforms existing methods.

Subfields: Artificial Intelligence, Optimization & Training, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Safety, Robustness & Interpretability

Citations in the corpus

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (2019, background): “…has focused on the important challenges of robustness to adversarial examples (Szegedy et al., 2014; Carlini & Wagner, 2017; 2016), unknown unknowns (Hendrycks et al., 2019; Hendrycks & Gimpel, 2017b; Liu et al., 2018), and model or data poisoning (Steinhardt et al., 2017; Hendrycks et al., 2018).”
On Robustness of Neural Architecture Search Under Label Noise (2020, background): “• Hierarchical noise (Hendrycks et al., 2018): All label classes can uniformly turn to any other label classes that belong to the same “superclass.””
Using Pre-Training Can Improve Model Robustness and Uncertainty (2019, methodology): “We also consider the Gold Loss Correction (GLC) method of Hendrycks et al. (2018), which assumes access to a small, trusted dataset of cleanly labeled (gold standard) examples, which is also known as a semi-veriﬁed setting (Charikar et al., 2017).”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, methodology): “There have been multiple lines of work to learn amidst noisy labels such as loss regularization [84, 96, 31], data re-weighting [68, 72], label cleaning [64, 42, 61], and”
Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty (2019, methodology): “In fact, auxiliary rotation prediction with no loss correction outperforms the GLC with 5% trusted data on CIFAR-100.”
Learning to Reweight Examples for Robust Deep Learning (2018, methodology): “…community (Natarajan et al., 2013; Angluin & Laird, 1988) and practical methods have also been proposed (Reed et al., 2014; Sukhbaatar & Fergus, 2014; Xiao et al., 2015; Azadi et al., 2016; Goldberger & Ben-Reuven, 2017; Li et al., 2017; Jiang et al., 2017; Vahdat, 2017; Hendrycks et al., 2018).”
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels (2018, methodology): “Several methods have been proposed to estimate it [11, 13, 31, 16].”
Unsupervised label noise modeling and loss correction (2019, methodology): “Other approaches modify class probabilities (Patrini et al., 2017; Hendrycks et al., 2018) by estimating the noise associated with each class, thus computing a loss that guides the training process towards the correct classes.”
Robust Multi-View Clustering With Incomplete Information (2022, methodology): “Based on the way of achieving robustness, the existing works could be roughly grouped into four categories, namely, i) robust loss based methods [30], [32], which design a loss function which is tolerant to noisy labels; ii) robust architecture based methods [33], [34], which modify the network arch”
A Comprehensive Survey on Poisoning Attacks and Countermeasures in Machine Learning (2022, methodology): “[48] proposed a loss correction method by utilizing information from a trusted small dataset.”
Confident Learning: Estimating Uncertainty in Dataset Labels (2019, background): “, 2018), semi-supervised learning (Hendrycks et al., 2018; Li et al., 2017b; Vahdat, 2017), symmetric cross entropy (Wang et al.”
Combating Label Noise in Deep Learning Using Abstention (2019, background): “…we do not need to estimate the label confusion matrix as in (Sukhbaatar et al., 2014; Reed et al., 2014; Patrini et al., 2017) or make assumptions regarding the amount of label noise or the existence of a trusted or clean data set as done in (Hendrycks et al., 2018) and (Li et al., 2017b).”
Unifying semi-supervised and robust learning by mixup (2019, methodology): “Some work of RLL uses p > 0 setting explicitly (Charikar et al. (2017); Li et al. (2017); Hendrycks et al. (2018)) and implicitly (Jiang et al. (2018); Jenni & Favaro (2018)).”
Computer-Aided Diagnosis in the Era of Deep Learning (2020, methodology): “Another study 49 proposed a loss correction technique that used a small data set with trusted labels to estimate the noise distribution of the label noise and showed that the method could improve the robustness of the deep networks for several vision and natural language processing tasks.”
Synthetic vs Real: Deep Learning on Controlled Noise (2019, background): “…2017), example weighting (Jiang et al., 2018; Ren et al., 2018; Shu et al., 2019; Jiang et al., 2015; Liang et al., 2016), semi-supervised learning (Hendrycks et al., 2018; Vahdat, 2017), data augmentation (Zhang et al., 2018; Cheng et al., 2019 et al., 2015; Li et al., 2017a; Krause et al.,…”
Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations (2018, background): “Most work on robustness in deep learning methods for vision has focused on the important challenges of robustness to adversarial examples [53, 6, 7], unknown unknowns [20, 41, 22], and model or data poisoning [4, 52, 23].”
Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning (2019, background): “ by embedding domain adaptation in the learning pipeline. Other approaches focus exclusively on learning useful representations from scratch in a target domain when annotation constraints are relaxed [23, 9, 5]. Semi-supervised learning [23] focuses on scenarios with sparsely labeled data and extens”
Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels (2019, background): “…Ren et al., 2018; Shu et al., 2019; Jiang et al., 2015; Liang et al., 2016), cross-validation (Northcutt et al., 2019), semi-supervised learning (Hendrycks et al., 2018; Vahdat, 2017; Li et al., 2020; Zhang et al., 2020), data augmentation (Zhang et al., 2018; Cheng et al., 2019; 2020; Liang et…”
Identifying Mislabeled Data using the Area Under the Margin Ranking (2020, methodology): “The trusted data can be used to estimate the noise-transition matrix for mislabeled data (Sukhbaatar et al., 2015; Hendrycks et al., 2018), or can be used to perform meta-learning (Ren et al., 2018), distillation (Li et al., 2017b), or pseudo-labeling (Zhang et al., 2019).”
Tilted Empirical Risk Minimization (2020, methodology): “For a fair comparison, we do not compare with approaches that require additional clean validation data [e.g., 21, 50, 54, 63], as such data can be costly to obtain in practice.”