Learning From Noisy Labels With Deep Neural Networks: A Survey

By Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, Jae-Gil Lee. 2020

1,409 citations recorded by the source corpus.

Deep learning has achieved remarkable success in numerous domains with help from large amounts of big data. However, the quality of data labels is a concern because of the lack of high-quality labels in many real-world scenarios. As noisy labels severely degrade the generalization performance of deep neural networks, learning from noisy labels (robust training) is becoming an important task in modern deep learning applications. In this survey, we first describe the problem of learning with label noise from a supervised learning perspective. Next, we provide a comprehensive review of 62 state-of-the-art robust training methods, all of which are categorized into five groups according to their methodological difference, followed by a systematic comparison of six properties used to evaluate their superiority. Subsequently, we perform an in-depth analysis of noise rate estimation and summarize the typically used evaluation methodology, including public noisy datasets and evaluation metrics. Finally, we present several promising research directions that can serve as a guideline for future studies.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “Two possible techniques are: identification via cross-validation (Song et al., 2022) (e.”
Online Continual Learning on a Contaminated Data Stream with Blurry Task Boundaries (2022, background): “According to [34], estimated noise ratio of both datasets are 20.”
Federated learning for medical image analysis: A survey (2023, background): “Learning from noisy labels (Song et al., 2022; Frénay and Verleysen, 2013) is the corresponding technique.”
Stacked Autoencoders Driven by Semi-Supervised Learning for Building Extraction from near Infrared Remote Sensing Imagery (2021, background): “Typically, training of a deep auto-encoder consists of two steps: (a) training per layer and (b) ﬁne tuning of the entire network [37].”
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023, background): “There is also a rich literature on the related problem of learning from noisy labels (Song et al., 2022).”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, background): “Therefore, a separate topic of study [44, 60, 60] emerged, which aims at learning robust models even from the noisy data distribution.”
Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases (2021, background): “by 16% (similar results shown for general NLP tasks (Song et al., 2022)) further showcasing the robustness of the models towards wrong labels.”
Robust Learning by Self-Transition for Handling Noisy Labels (2020, background): “3467222 such as Amazon’s Mechanical Turk and surrounding tags of collected data, have been widely used to mitigate the high labeling cost, but they often yield samples with noisy labels that may not be true [35].”
Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls (2022, background): “Compared to much of the literature on learning with noisy inputs (Song et al., 2020) two distinguishing factors of this setting are that undesirable data from deployment is dependent on the user, and that noise is not only random but can also be adversarial.”
A Survey on Programmatic Weak Supervision (2022, methodology): “Apart from the above methods, other approaches designed for learning with noisy labels [Song et al. , 2020] can also be utilized as end models.”
ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022, methodology): “Reciprocally, methods that learn from noisy labels can be integrated into the training of the tiny task model (Song et al., 2020).”
DualGraph: A graph-based method for reasoning about label noise (2021, methodology): “To avoid any false corrections, many recent studies [7, 11, 17, 36] have adopted sample selection that involves selecting true-labeled samples from a noisy training dataset [27].”
Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation (2021, methodology): “Therefore, this work is also motivated by emerging techniques [52] of learning from noisy labels.”
Robust Multi-View Clustering With Incomplete Information (2022, background): “In recent years, a series of works [27], [28], [29], [30], [31] have been conducted to endow neural networks with the robustness against noisy labels.”