Toward Robustness against Label Noise in Training Deep Discriminative Neural Networks

By Arash Vahdat. 2017

318 citations recorded by the source corpus.

Collecting large training datasets, annotated with high-quality labels, is costly and time-consuming. This paper proposes a novel framework for training deep convolutional neural networks from noisy labeled datasets that can be obtained cheaply. The problem is formulated using an undirected graphical model that represents the relationship between noisy and clean labels, trained in a semi-supervised setting. In our formulation, the inference over latent clean labels is tractable and is regularized during training using auxiliary sources of information. The proposed model is applied to the image labeling problem and is shown to be effective in labeling unseen images as well as reducing label noise in training on CIFAR-10 and MS COCO datasets.

Subfields: Artificial Intelligence, Civil and Structural Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Civil and Structural Engineering

Citations in the corpus

On Robustness of Neural Architecture Search Under Label Noise (2020, methodology): “Another group of approaches treats the true but unknown labels as latent variables and the noisy labels as observed variables so that EMlike algorithms can be used to learn the true label distribution of the dataset (Xiao et al., 2015; Vahdat, 2017; Khetan et al., 2018).”
Learning to Reweight Examples for Robust Deep Learning (2018, methodology): “…community (Natarajan et al., 2013; Angluin & Laird, 1988) and practical methods have also been proposed (Reed et al., 2014; Sukhbaatar & Fergus, 2014; Xiao et al., 2015; Azadi et al., 2016; Goldberger & Ben-Reuven, 2017; Li et al., 2017; Jiang et al., 2017; Vahdat, 2017; Hendrycks et al., 2018).”
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels (2018, background): “Other approaches include treating the true labels as a latent variable and the noisy labels as an observed variable so that EM-like algorithms can be used to learn true label distribution of the dataset [40, 17, 36].”
Unsupervised label noise modeling and loss correction (2019, background): “Other approaches seek to relabel the noisy samples by modeling their noise through directed graphical models (Xiao et al., 2015), Conditional Random Fields (Vahdat, 2017), or CNNs (Veit et al., 2017).”
MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels (2017, methodology): “More recently, methods utilized additional labels for label cleaning (Veit et al., 2017), knowledge distillation (Li et al., 2017b) or semi-supervised learning (Vahdat, 2017; Dehghani et al., 2017).”
NLNL: Negative Learning for Noisy Labels (2019, methodology): “Loss correction [25, 31, 36, 10] methods assume that the noise transition matrix is already known or that some clean data are obtainable to calculate the noise transition matrix.”
Joint Optimization Framework for Learning with Noisy Labels (2018, methodology): “Conversely, we need only the prior distribution p, and thus our condition is more general than that of [14, 19].”
Synthetic vs Real: Deep Learning on Controlled Noise (2019, background): “…noise), in which the label of each example is independently and uniformly changed to a random (incorrect) class with a probability (Zhang et al., 2017; Arpit et al., 2017; Vahdat, 2017; Shu et al., 2019; Jiang et al., 2018; Ma et al., 2018; Han et al., 2018; Li et al., 2019; Arazo et al., 2019).”
Graph Convolutional Label Noise Cleaner: Train a Plug-And-Play Action Classifier for Anomaly Detection (2019, methodology): “In the case of noise reduction, they aim to correct noisy labels via formulating the noise model explicitly or implicitly, such as Conditional Random Fields (CRF) [60], knowledge graphs [37].”
Part-dependent Label Noise: Towards Instance-dependent Label Noise (2020, background): “Learning with noisy labels can be dated back to [4], which has recently drawn a lot of attention, especially from the deep learning community, e.g., [52, 79, 25, 13, 50, 57, 77, 37, 70, 75, 18, 41, 53, 22, 40, 56, 17, 16, 61, 60, 33, 32, 31, 21, 39, 46, 72, 66, 78].”
Symmetric Cross Entropy for Robust Learning With Noisy Labels (2019, background): “One common approach is to correct noisy labels to their true labels via a clean label inference step using complex noise models characterized by directed graphical models [26], conditional random fields [23], neural networks [11, 24] or knowledge graphs [12].”
Are Anchor Points Really Indispensable in Label-Noise Learning? (2019, background): “For example, many state-of-the-art approaches in this category are speciﬁcally designed to, e.g., select reliable examples [45, 14, 24], reweight examples [33, 15], correct labels [23, 17, 37, 32], employ side information [39, 21], and (implicitly) add regularization [13, 12, 43, 39, 21].”
Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels (2019, background): “…et al., 2019; Jiang et al., 2015; Liang et al., 2016), cross-validation (Northcutt et al., 2019), semi-supervised learning (Hendrycks et al., 2018; Vahdat, 2017; Li et al., 2020; Zhang et al., 2020), data augmentation (Zhang et al., 2018; Cheng et al., 2019; 2020; Liang et al., 2020), among…”
MoPro: Webly Supervised Learning with Momentum Prototypes (2020, methodology): “, 2019; 2020a), or require access to a set of samples with clean labels (Vahdat, 2017; Veit et al., 2017; Lee et al., 2018), which limit their scalability in practice.”
Human Uncertainty Makes Classification More Robust (2019, background): “Finally, although there has been much work scaling the number of images in datasets [18], and investigating label noise [40, 12, 48], little effort has been put into identifying the benefits from increasing the richness of (informative) label distributions for image classification tasks.”
Learning to Learn From Noisy Labeled Data (2018, methodology): “One line of approaches formulate explicit or implicit noise models to characterize the distribution of noisy and true labels, using neural networks [5, 8, 11, 19, 16, 23, 29], directed graphical models [31], knowledge graphs [13], or conditional random fields [27].”
L_DMI: An Information-theoretic Noise-robust Loss Function (2019, unknown): “Some of them focused on estimating the noise transition matrix to handle the label noise and proposed a variety of ways to constrain the optimization [8,9,37,39,43,44].”
Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification (2020, methodology): “Loss correction methods (Patrini et al., 2017; Vahdat, 2017; Xiao et al., 2015) tried to model the noise transition matrix, however, such matrix is hard to estimate in real-world tasks, e . g . unsupervised person re-ID with noisy pseudo labels obtained via clustering algorithm.”