Unsupervised label noise modeling and loss correction

By Eric Arazo Sanchez, Diego Ortego, Paul Albert, Noel Edward O'Connor, Kevin McGuinness. 2019

750 citations recorded by the source corpus.

Despite being robust to small amounts of label noise, convolutional neural networks trained with stochastic gradient methods have been shown to easily fit random labels. When there are a mixture of correct and mislabelled targets, networks tend to fit the former before the latter. This suggests using a suitable two-component mixture model as an unsupervised generative model of sample loss values during training to allow online estimation of the probability that a sample is mislabelled. Specifically, we propose a beta mixture to estimate this probability and correct the loss by relying on the network prediction (the so-called bootstrapping loss). We further adapt mixup augmentation to drive our approach a step further. Experiments on CIFAR-10/100 and TinyImageNet demonstrate a robustness to label noise that substantially outperforms recent state-of-the-art. Source code is available at https://git.io/fjsvE and Appendix at https://arxiv.org/abs/1904.11238.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Signal Processing

Citations in the corpus

Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “Other approaches consider training loss (Han et al., 2018; Arazo et al., 2019; Shen & Sanghavi, 2019), confidence (Hovy et al., 2013), prediction variance (Chang et al., 2017), and area under the curve (Pleiss et al., 2020).”
Online Continual Learning on a Contaminated Data Stream with Blurry Task Boundaries (2022, methodology): “Specifically, we first split the examples in the episodic memory into clean and noisy subsets based on the smallloss trick that is widely used when handling noisy labels [3, 22].”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, background): “Several works design the noise corrected losses [86, 28, 46, 4, 84] so that the loss minimization of the whole data becomes similar to that of clean samples.”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, methodology): “Other approaches have used training loss (Han et al., 2018; Arazo et al., 2019; Shen and Sanghavi, 2019), confidence (Hovy et al.”
Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks (2019, background): “(2018); Arazo et al. (2019) for additional work on dealing with label noise in classification tasks.”
Understanding and Improving Early Stopping for Learning with Noisy Labels (2021, methodology): “M-correction [1] uses two Gaussian Mixture Models to identify conﬁdent examples, instead of using networks themselves.”
Delving Deep Into Label Smoothing (2020, neutral): “We follow the same experimental settings as in [49], [51].”
Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels (2019, background): “…label is also commonly used as a benchmark to evaluate robust learning methods in a noise-control setting e.g. in (Vahdat, 2017; Shu et al., 2019; Jiang et al., 2018; Ma et al., 2018; Han et al., 2018b; Van Rooyen et al., 2015; Li et al., 2019; Arazo et al., 2019; Charoenphakdee et al., 2019).”
DivideMix: Learning with Noisy Labels as Semi-supervised Learning (2020, methodology): “3 (Arazo et al., 2019) Last 93.”
Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning (2019, background): “Other approaches focus exclusively on learning useful representations from scratch in a target domain when annotation constraints are relaxed [6, 9, 10].”
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023, methodology): “This loss is related to the bootstrapping methods in Reed et al. (2014) and Arazo et al. (2019) for addressing label noise.”
Synthetic vs Real: Deep Learning on Controlled Noise (2019, background): “Following (Jiang et al., 2018; Shu et al., 2019; Arazo et al., 2019), the noisy CIFAR-100 data are of uniform label-ﬂipping noise, where the label of each image is independently changed to a uniform (incorrect) class with probability p , where p is the noise level and is set to 20%, 40%, 60%, and…”
Noise against noise: stochastic label noise helps combat inherent label noise (2021, methodology): “Many methods, demonstrated to work well, select or add higher weight on small-loss samples (Han et al., 2018b; Jiang et al., 2018; Li et al., 2020), or use the model’s predictions to correct noisy labels (Tanaka et al., 2018; Arazo et al., 2019).”
Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise (2020, positive): “…and Mitchell 1998; Yan et al. 2017; Patrini et al. 2016; Zhang and Sabuncu 2018; Xu et al. 2019; Menon et al. 2020; Ma et al. 2020) or inexplicitly used in experiments for synthetizing noisy labels (Han et al. 2018b; Yu et al. 2019; Arazo et al. 2019; Li, Socher, and Hoi 2020; Lukasik et al. 2020).”
Improving Generalization by Controlling Label-Noise Information in Neural Network Weights (2020, neutral): “, 2019; Arazo et al., 2019). Han et al. (2018); Yu et al.”
Unsupervised Person Re-Identification via Multi-Label Classification (2020, methodology): “Inspired by (Arazo et al., 2019; Tanaka et al., 2018),we aim to push the probability on each positive label to 1/|P∗ i |, where |P∗ i | is the number of positive classes of xi .”
Early-Learning Regularization Prevents Memorization of Noisy Labels (2020, background): “initialize averaged weights (untrainable) for t in [1,num_epochs] do for k in [1, 2] do .”