Combating Label Noise in Deep Learning Using Abstention

By Sunil Thulasidasan, Tanmoy Bhattacharya, J. Bilmes, Gopinath Chennupati, J. Mohd-Yusof. 2019

196 citations recorded by the source corpus.

We introduce a novel method to combat label noise when training deep neural networks for classification. We propose a loss function that permits abstention during training thereby allowing the DNN to abstain on confusing samples while continuing to learn and improve classification performance on the non-abstained samples. We show how such a deep abstaining classifier (DAC) can be used for robust learning in the presence of different types of label noise. In the case of structured or systematic label noise -- where noisy training labels or confusing examples are correlated with underlying features of the data-- training with abstention enables representation learning for features that are associated with unreliable labels. In the case of unstructured (arbitrary) label noise, abstention during training enables the DAC to be used as an effective data cleaner by identifying samples that are likely to have label noise. We provide analytical results on the loss function behavior that enable dynamic adaption of abstention rates based on learning progress during training. We demonstrate the utility of the deep abstaining classifier for various image classification tasks under different types of label noise; in the case of arbitrary label noise, we show significant improvements over previously published results on multiple image benchmarks. Source code is available at this https URL

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, methodology): “Our approach, DisCor, can be interpreted as only indirectly affecting generalization via the target Q-values for state-action pairs that will be used as bootstrap targets for the Bellman backup, which are expected to be accurate with DisCor, and this can aid generalization, similar to how generaliza”
Learning From Noisy Anchors for One-Stage Object Detection (2019, background): “It has been demonstrated that the outputs from networks can indicate the noise level of samples when labels are corrupted and noisy for image classification tasks—the network tends to learn clean samples quickly early on and make confident predictions for them, while recognizing noisy samples slowly”
SELF: Learning to Filter Noisy Labels with Self-Ensembling (2019, background): “(Ren et al., 2018) and DAC (Thulasidasan et al., 2019) considered the setting of having a small clean validation set of 1000 and 5000 images respectively.”
DivideMix: Learning with Noisy Labels as Semi-supervised Learning (2020, background): “The second type of correction focuses on reweighting training samples or separating clean and noisy samples, which results in correcting the loss function (Thulasidasan et al., 2019; Konstantinov & Lampert, 2019).”
Deep learning with noisy labels: exploring techniques and remedies in medical image analysis (2019, background): “[135] proposed modifying the cross-entropy loss function”
An Overview of Deep Semi-Supervised Learning (2020, background): “Another type of correction applies a reweighing to the training examples to distinguish the clean and noisy samples [26, 147].”
Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise (2020, unknown): “Thulasidasan et al. (2019) investigate the co-occurrence of noisy labels with underlying features by adding synthetic features, such as the smudge, to mislabeled instances.”
Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey (2019, background): “samples and clean samples. On top of that, not to miss valuable information from minority and hard samples, noisy data is weighted according to their noisiness level which is estimated by pLOF [140]. [86] introduces abstention, which gives option to abstain samples, depending on their crossentropy e”
Robust early-learning: Hindering the memorization of noisy labels (2021, background): “These settings are widely used in existing works (Ma et al., 2018; Thulasidasan et al., 2019; Pleiss et al., 2020; Wang et al., 2021a).”
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics (2022, background): “…of data properties, or metadata which different examples might exhibit, including but not limited to: noisy (Wu et al., 2020; Yi and Wu, 2019; Thulasidasan et al., 2019a;b), atypical (Hooker et al., 2020; Buolamwini and Gebru, 2018; Hashimoto et al., 2018; S(cid:32)lowik and Bottou, 2021),…”
On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks (2019, background): “Further mixup based models can also be combined with rejection classifiers, both during training [31] for dealing with label noise, as well as inference [7] to improve the training and classification pipeline in modern deep learning.”
Does label smoothing mitigate label noise? (2020, background): “Indeed, label smoothing is conspicuously absent in most treatments of the noisy label problem [Patrini et al., 2016, Han et al., 2018b, Charoenphakdee et al., 2019, Thulasidasan et al., 2019, Amid et al., 2019].”