Training Deep Neural Networks on Noisy Labels with Bootstrapping

By Scott E. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, Andrew Rabinovich. 2014

1,110 citations recorded by the source corpus.

Current state-of-the-art deep learning systems for visual object recognition and detection use purely supervised training with regularization such as dropout to avoid overfitting. The performance depends critically on the amount of labeled examples, and in current practice the labels are assumed to be unambiguous and accurate. However, this assumption often does not hold; e.g. in recognition, class labels may be missing; in detection, objects in the image may not be localized; and in general, the labeling may be subjective. In this work we propose a generic way to handle noisy and incomplete labeling by augmenting the prediction objective with a notion of consistency. We consider a prediction consistent if the same prediction is made given similar percepts, where the notion of similarity is between deep network features computed from the input data. In experiments we demonstrate that our approach yields substantial robustness to label noise on several datasets. On MNIST handwritten digits, we show that our model is robust to label corruption. On the Toronto Face Database, we show that our model handles well the case of subjective labels in emotion recognition, achieving state-of-the- art results, and can also benefit from unlabeled face images with no modification to our method. On the ILSVRC2014 detection challenge data, we show that our approach extends to very deep networks, high resolution images and structured outputs, and results in improved scalable detection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Safety, Robustness & Interpretability

Citations in the corpus

Webly Supervised Learning of Convolutional Networks (2015, methodology): “The outline of our approach is shown in Figure 2.”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “Two possible techniques are: identification via cross-validation (Song et al., 2022) (e.g., finding examples where predictions disagree with human labels), or via a modification of the loss function called bootstrapping (Reed et al., 2014).”
Toward Controlled Generation of Text (2017, methodology): “To alleviate the issue of noisy data and ensure robustness of model optimization, we incorporate a minimum entropy regularization term (Grandvalet et al., 2004; Reed et al., 2014).”
Scalable, High-Quality Object Detection (2014, background): “2 (found by cross-validation on the maximum overlap of priors with the groundtruth boxes), and where the image region is normalized to the unit box.”
Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels (2015, background): “Bootstrapping [40], semi-supervised learning (SSL) [45, 57] and Positive Unlabeled (PU) learning [10, 28, 30, 34] are other ways of learning from noisy labeled data.”
Regularizing Neural Networks by Penalizing Confident Output Distributions (2017, background): “Instead of smoothing the labels with a uniform distribution, as in label smoothing, we can smooth the labels with a teacher model (Hinton et al., 2015) or the model’s own distribution (Reed et al., 2014).”
Logic Tensor Networks for Semantic Image Interpretation (2017, background): “Very large visual recognition datasets now exist which are noisy [Reed et al., 2014].”
The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition (2015, methodology): “Some have shown that reasoning about label noise can have a substantial effect on recognition performance [58], while others demonstrate only small improvements by reducing the noise level or having a noise-aware model [47, 38, 52].”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, methodology): “[56, 62] relabel the samples using the model predictions.”
Data Distillation: Towards Omni-Supervised Learning (2017, methodology): “[29] use a reconstruction consistency term for training classification and detection models.”
Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition (2019, background): “While novel changes to architectures have been proposed [57, 61] to combat label noise, our experiments demonstrate that large-scale training of an existing video architecture [14] makes it noise-resilient.”
Label Refinery: Improving ImageNet Classification through Label Progression (2018, methodology): “[12] augments noisy labels using other models to improve label consistency.”
Co-teaching: Robust training of deep neural networks with extremely noisy labels (2018, methodology): “MNIST, CIFAR10 and CIFAR-100 are used here (Table 1), because these data sets are popularly used for evaluation of noisy labels in the literature [13, 31, 33].”
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels (2018, background): “Alternatively, one could gradually replace noisy labels by neural network predictions [32, 35].”
Learning from Noisy Labels with Distillation (2017, methodology): “The Bootstrapping method [16] and Label Smoothing [19] are popular approaches to reduce the noise effects.”