Learning from Noisy Labels with Deep Neural Networks

By Sainbayar Sukhbaatar, Rob Fergus. 2014

280 citations recorded by the source corpus.

The availability of large labeled datasets has allowed Convolutional Network models to achieve impressive recognition results. However, in many settings manual annotation of the data is impractical; instead our data has noisy labels, i.e. there is some freely available label for each image which may or may not be accurate. In this paper, we explore the performance of discriminatively-trained Convnets when trained on such noisy data. We introduce an extra noise layer into the network which adapts the network outputs to match the noisy label distribution. The parameters of this noise layer can be estimated as part of the training process and involve simple modifications to current training infrastructures for deep networks. We demonstrate the approaches on several datasets, including large scale experiments on the ImageNet classification benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “In such cases, we plan to improve our method by designing loss functions or layers specific to noise reduction as in [48], providing a more principled way for learning the multi-modal embedding in presence of significant noise.”
Webly Supervised Learning of Convolutional Networks (2015, methodology): “The outline of our approach is shown in Figure 2.”
Residual Attention Network for Image Classification (2017, methodology): “In this experiment, we show our Residual Attention Network enjoys noise resistant property on CIFAR-10 dataset following the setting of paper [31].”
The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition (2015, methodology): “Some have shown that reasoning about label noise can have a substantial effect on recognition performance [58], while others demonstrate only small improvements by reducing the noise level or having a noise-aware model [47, 38, 52].”
Supervised Contrastive Learning (2020, background): “A number of papers have studied other drawbacks of the cross-entropy loss, such as sensitivity to noisy labels [63, 46], presence of adversarial examples [10, 36], and poor margins [2].”
An Empirical Study of Example Forgetting during Deep Neural Network Learning (2018, background): “Second, we aim to characterize whether forgetting statistics can be used to identify “important” samples and detect outliers and examples with noisy labels (John, 1995; Brodley & Friedl, 1999; Sukhbaatar et al., 2014; Jiang et al., 2018).”
Training Deep Neural Networks on Noisy Labels with Bootstrapping (2014, background): “Sukhbaatar & Fergus (2014) developed two deep learning techniques for handling noisy labels, learning to model the noise distribution in a top-down and bottom-up fashion.”
Learning to Reweight Examples for Robust Deep Learning (2018, methodology): “…by the learning theory community (Natarajan et al., 2013; Angluin & Laird, 1988) and practical methods have also been proposed (Reed et al., 2014; Sukhbaatar & Fergus, 2014; Xiao et al., 2015; Azadi et al., 2016; Goldberger & Ben-Reuven, 2017; Li et al., 2017; Jiang et al., 2017; Vahdat, 2017;…”
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels (2018, background): “Firstly, Sukhbaatar and Fergus [35] proposed accounting for noisy labels with a confusion matrix so that the cross entropy loss becomes where c represents number of classes, (cid:101) y represents noisy labels, y represents the latent true labels and p ( (cid:101) y = (cid:101) y n | y = i ) is the ”
Learning from Noisy Labels with Distillation (2017, methodology): “Most existing work use simulated approaches to evaluate their method on learning from label noise [12, 18], where they inject label noise based on some controlled and known corruption process to the clean dataset.”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, methodology): “…(a) formulating explicit or implicit noise models to characterize the distribution of noisy and true labels using neural networks (Goldberger and BenReuven 2017; Jiang et al. 2018; Sukhbaatar et al. 2015), graphical models (Xiao et al. 2015; Li et al. 2017), etc. and (b) using correction methods.”
Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise (2018, methodology): “Now with Ĉ, we follow the method of [23, 16] to train a corrected classifier, which we now briefly describe.”
Cleaning Crowdsourced Labels Using Oracles For Statistical Classification (2018, background): “[37] train neural networks on images with noisy labels directly, but add an extra layer designed to model the label noise.”
Suppressing Uncertainties for Large-Scale Facial Expression Recognition (2020, background): “In order to handle noisy labels, one intuitive idea is to leverage a small set of clean data that can be used to assess the quality of the labels during the training process [40, 23, 8], or to estimate the noise distribution [36], or to train the feature extractors [3].”
Harnessing noisy Web images for deep representation (2017, background): “In Sukhbaatar et al. (2014) and soon followed by Xiao et al. (2015) several solutions to train deep convolutional networks as classifiers under noisy condition are studied.”
Collaborative Unsupervised Domain Adaptation for Medical Image Diagnosis (2019, methodology): “Based on the collected dataset3, we randomly choose part of normal cases and all typical pneumonia cases to make up the source domain, and use the rest of normal cases and all COVID-19 cases as the target domain [12].”
CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images (2018, background): “Another so-lution is to develop noise-cleaning methods that aim to remove or correct the mislabelled examples in training data [32].”
Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach (2016, background): “The second, inspired by [39], is named “ forward ” as it multiplies the network predictions by T .”
Deep learning with noisy labels: exploring techniques and remedies in medical image analysis (2019, methodology): “most applications, including segmentation (experiments in Section V-C; [103]; [104]; [105]), lesion detection (experiments in Section V-A), and classification ([106]) [107] - image classification [76] - image classification [108] - image classification [109] - image classification [110] - boundary s”
Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting (2019, background): “For instance, practically collected training samples always contain corrupted labels [10, 11, 12, 13, 14, 15, 16, 17].”