Does label smoothing mitigate label noise?

By Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, Surinder Kumar. 2020

424 citations recorded by the source corpus.

Label smoothing is commonly used in training deep learning models, wherein one-hot training labels are mixed with uniform label vectors. Empirically, smoothing has been shown to improve both predictive performance and model calibration. In this paper, we study whether label smoothing is also effective as a means of coping with label noise. While label smoothing apparently amplifies this problem --- being equivalent to injecting symmetric noise to the labels --- we show how it relates to a general family of loss-correction techniques from the label noise literature. Building on this connection, we show that label smoothing is competitive with loss-correction under label noise. Further, we show that when distilling models from noisy data, label smoothing of the teacher is beneficial; this is in contrast to recent findings for noise-free problems, and sheds further light on settings where label smoothing is beneficial.

Subfields: Artificial Intelligence, Civil and Structural Engineering, Computational Theory and Mathematics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Civil and Structural Engineering

In Computational Theory and Mathematics

In Optimization & Training

Citations in the corpus

Incremental Learning in Semantic Segmentation from Image Labels (2021, methodology): “For this reason, we propose to smooth the pseudo-labels to reduce the noise [42].”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, background): “[45] shows that using smooth labels is beneficial in noisy labeled data learning.”
Why Do Better Loss Functions Lead to Less Transferable Features? (2020, background): “Label smoothing has been explained in terms of mitigation of label noise [39, 13], entropy regularization [52, 41], and accelerated convergence [78].”
Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise (2020, neutral): “…and Mitchell 1998; Yan et al. 2017; Patrini et al. 2016; Zhang and Sabuncu 2018; Xu et al. 2019; Menon et al. 2020; Ma et al. 2020) or inexplicitly used in experiments for synthetizing noisy labels (Han et al. 2018b; Yu et al. 2019; Arazo et al. 2019; Li, Socher, and Hoi 2020; Lukasik et al. 2020).”
Noise against noise: stochastic label noise helps combat inherent label noise (2021, methodology): “On CIFAR-10 and CIFAR-100, we compare with the following baselines: 1) standard cross-entropy (CE) loss, 2) Generalized Cross-Entropy (GCE) (Zhang & Sabuncu, 2018) 2020) that clips the label to be less conﬁdent before training.”
Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study (2021, background): “Since LS has weight shrinkage and regularisation effects (Lukasik et al., 2020), we have following conjecture: Conjecture 1.”
Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation (2021, methodology): “However, the proper objective derived from both logit matching and label matching enables similar effects of label smoothing, as studied in [Lukasik et al. , 2020; Yuan et al. , 2020].”