When Does Label Smoothing Help?

By Rafael Müller, Simon Kornblith, Geoffrey E. Hinton. 2019

2,425 citations recorded by the source corpus.

The generalization and learning speed of a multi-class neural network can often be significantly improved by using soft targets that are a weighted average of the hard targets and the uniform distribution over labels. Smoothing the labels in this way prevents the network from becoming over-confident and label smoothing has been used in many state-of-the-art models, including image classification, language translation and speech recognition. Despite its widespread use, label smoothing is still poorly understood. Here we show empirically that in addition to improving generalization, label smoothing improves model calibration which can significantly improve beam-search. However, we also observe that if a teacher network is trained with label smoothing, knowledge distillation into a student network is much less effective. To explain these observations, we visualize how label smoothing changes the representations learned by the penultimate layer of the network. We show that label smoothing encourages the representations of training examples from the same class to group in tight clusters. This results in loss of information in the logits about resemblances between instances of different classes, which is necessary for distillation, but does not hurt generalization or calibration of the model's predictions.

Subfields: Artificial Intelligence, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Signal Processing

Citations in the corpus

Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “, 2020) and prevents overfitting to the training set (Müller et al., 2019), thus improving generalization to different domains.”
Knowledge Distillation: A Survey (2020, methodology): “nt kinds of knowledge help each other in a complementary manner. For example, the response-based knowledge has a similar motivation to label smoothing and the model regularization (Kim and Kim, 2017; Muller et al., 2019; Ding et al., 2019); The featuredbased knowledge is often used to mimic the inte”
Supervised Contrastive Learning (2020, positive): “Alternative losses have been proposed, but the most effective ideas in practice have been approaches that change the reference label distribution, such as label smoothing [47, 35], data augmentations such as Mixup [60] and CutMix [59], and knowledge distillation [21].”
Meta Pseudo Labels (2020, background): “Outside the line of work on distillation, manually designed soft labels, such as label smoothing [45] and temperature sharpening or dampening [76, 77], have also been shown to improve models’ generalization.”
Transformers without Tears: Improving the Normalization of Self-Attention (2019, methodology): “This corresponds to the known overconﬁdence of translation models’ log-its, on which label smoothing has a downscaling effect (M¨uller et al., 2019).”
Essentials for Class Incremental Learning (2021, background): “sentations to be closer to their respective class centroid and equidistant to the other class centroids [21], and this comes at the expense of inter-class sample relationships, i.”
Robust fine-tuning of zero-shot models (2021, methodology): “Hyperparameters for each regularization strategy are as follows: (i) The weight decay coefficient is set to 10−λ where λ is chosen uniformly at random from [0, 4] for each trial; (ii) The L1 regularization coefficient is set to 10−λ where λ is chosen uniformly at random from [4, 8] for each trial; (”
Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning (2020, background): “Effective alternatives have been proposed to change the reference label distributions through label smoothing (Szegedy et al., 2016; M¨uller et al., 2019), Mixup (Zhang et al., 2018), CutMix (Yun et al., 2019), knowledge distillation (Hinton et al., 2015) or self-training (Yalniz et al., 2019; Xie…”
Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation (2020, background): “Prior attempts in label smoothing (Szegedy et al., 2016; Vaswani et al., 2017; Chorowski and Jaitly, 2017; Müller et al., 2019) focus on categorical labels to prevent the network from becoming overconfident while we apply label smoothing on ordinal labels (i.e., Likert scores) to prevent the…”
Improving Calibration for Long-Tailed Recognition (2020, background): “It relieves over-fitting and increases calibration and reliability [23].”
Understanding and Improving Knowledge Distillation (2020, methodology): “However, recently Mirzadeh et al. (2019) and M¨uller et al. (2019) have found this intuition would fail under certain circum-stances.”
Class-Incremental Learning via Dual Augmentation (2021, methodology): “On the contrary, Mixup and Label-Smoothing (LS) [50] lead to significantly smaller eigenvalues for the several top eigenvectors, which represent more compact representations.”
More Grounded Image Captioning by Distilling Image-Text Matching Model (2020, background): “This is because their localizer and decoder are learned jointly and coupled loosely in the attention module, easily resulting in modal collapse [41].”
Delving Deep Into Label Smoothing (2020, neutral): “Although combining the uniform distribution with the original hard label is useful for regularization, LS itself does not consider the genuine relationships among different categories [55].”
Learning to Give Checkable Answers with Prover-Verifier Games (2021, neutral): “We employed a bit of Label Smoothing [24] (with coeﬃcient 0 .”
A Survey on Data Augmentation for Text Classification (2021, neutral): “Otherwise, methods like label smoothing [10] can model a general uncertainty.”
Beyond Noise: Mitigating the Impact of Fine-grained Semantic Divergences on Neural Machine Translation (2021, background): “M¨uller et al. (2019) investigate the effect of label smoothing on calibration.”