Deep Learning is Robust to Massive Label Noise

By David Rolnick, Andreas Veit, Serge J. Belongie, Nir Shavit. 2017

639 citations recorded by the source corpus.

Deep neural networks trained on large supervised datasets have led to impressive results in image classification and other tasks. However, well-annotated datasets can be time-consuming and expensive to collect, lending increased interest to larger but noisy datasets that are more easily obtained. In this paper, we show that deep neural networks are capable of generalizing from training data for which true labels are massively outnumbered by incorrect labels. We demonstrate remarkably high test performance after training on corrupted data from MNIST, CIFAR, and ImageNet. For example, on MNIST we obtain test accuracy above 90 percent even after each clean training example has been diluted with 100 randomly-labeled examples. Such behavior holds across multiple patterns of label noise, even when erroneous labels are biased towards confusing classes. We show that training in this regime requires a significant but manageable increase in dataset size that is related to the factor by which correct labels have been diluted. Finally, we provide an analysis of our results that shows how increasing noise decreases the effective batch size.

Subfields: Artificial Intelligence, Civil and Structural Engineering, Industrial and Manufacturing Engineering, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Civil and Structural Engineering

In Industrial and Manufacturing Engineering

In Safety, Robustness & Interpretability

Citations in the corpus

WoodScape: A Multi-Task, Multi-Camera Fisheye Dataset for Autonomous Driving (2019, background): “unately, due to the overwhelming success of deep learning, recently it does not get as much attention as it still deserves in our opinion. Deep learning was shown to be quite resistant to label noise [41]. However, at the same time, it was shown that careful inspection of the training sets for outli”
What do we need to build explainable AI systems for the medical domain? (2017, neutral): “This has led to a proliferation of deep learning results on the same tasks using the same well-known data sets (Rolnick et al., 2017).”
Are Labels Required for Improving Adversarial Robustness? (2019, background): “In particular, while standard training procedures can be robust to random label noise, as observed in previous work [29, 31], accuracy decreases almost one-to-one (slope -0.”
EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks (2019, background): “But even so, “deep learning is robust to massive label noise” (Rolnick et al., 2017).”
The iNaturalist Challenge 2017 Dataset (2017, background): “While current deep models are robust to label noise at training time, it is still very important to have clean validation and test sets to be able to quantify performance [38, 30].”
A cloud detection algorithm for satellite imagery based on deep learning (2019, background): “Deep learning models have shown to be resilient to noisy training data, and deep neural networks are capable of generalizing even though there is a majority of incorrect labels in the training dataset (Rolnick et al., 2017).”
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning (2018, background): “different Heavy-Tailed Universality classes [46, 32, 18, 25, 20, 5, 111, 7, 38, 17, 90, 8, 99]; other work on RMT approaches [133, 120, 114, 112, 87, 87, 137, 85, 126]; other work on statistical physics approaches [132, 52, 117, 125, 137, 119, 113]; work on fitting to noisy versus reliable signal [1”
Synthetic vs Real: Deep Learning on Controlled Noise (2019, background): “For example, Rolnick et al. (2017) experimented on a slightly different type of uniform noise and surprisingly found that DNNs are robust to massive label noise.”
Poisoning Web-Scale Training Datasets is Practical (2023, background): “This quantity-over-quality tradeoff has so far been deemed acceptable, both because modern neural networks are extremely resilient to large amounts of label noise [55, 83], and because”
CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise (2017, methodology): “Some methods were developed for directly learning neural network with label noise [1, 3, 14, 20, 22, 27, 32, 35, 41].”
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild (2018, background): “The second is a broader assumption, namely that deep CNNs are highly effective at training on large, noisy datasets (this was recently explored in [45, 59], who showed that despite the presence of high label noise, very strong features can be learned on large datasets).”
Choosing the Sample with Lowest Loss makes SGD Robust (2020, background): “[Rolnick et al., 2017] showed that deep neural networks are robust to random label noise especially for datasets like MNIST and CIFAR10.”
The Devil of Face Recognition is in the Noise (2018, background): “3) Outliers seem to have a less abrupt eﬀect on the performance across all losses, matching the observation in [10] and [17].”
A clinical text classification paradigm using weak supervision and deep representation (2019, result): “Our study shows that deep neural networks are robust to massive label noise and that sufficiently large training data is important for effectively training deep neural networks, which is consistent with a recent study in the common machine learning domain [71].”