Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study

By Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang-Ting Cheng, Marios Savvides. 2021

83 citations recorded by the source corpus.

This work aims to empirically clarify a recently discovered perspective that label smoothing is incompatible with knowledge distillation. We begin by introducing the motivation behind on how this incompatibility is raised, i.e., label smoothing erases relative information between teacher logits. We provide a novel connection on how label smoothing affects distributions of semantically similar and dissimilar classes. Then we propose a metric to quantitatively measure the degree of erased information in sample's representation. After that, we study its one-sidedness and imperfection of the incompatibility view through massive analyses, visualizations and comprehensive experiments on Image Classification, Binary Networks, and Neural Machine Translation. Finally, we broadly discuss several circumstances wherein label smoothing will indeed lose its effectiveness. Project page: http://zhiqiangshen.com/projects/LS_and_KD/index.html.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Eric P. Xing
2. Zhiqiang Shen
3. Hongsheng Li
4. Lu Yuan
5. Jianfeng Gao
6. Bin Xiao
7. Marios Savvides
8. Jianlong Fu

In Optimization & Training

1. Zhiqiang Shen
2. Marios Savvides
3. Eric P. Xing
4. Zechun Liu
5. Lu Yuan
6. Dejia Xu
7. Trevor Darrell
8. Bin Xiao

Citations in the corpus

• Un-mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning (2020, background): “Interestingly, it can be observed from visualizations of previous literature [M ¨ uller, Kornblith, and Hinton 2019, Shen et al. 2021] that label smoothing tends to force the output prediction of networks being less confident ( i.e. , lower maximum probability of predictions), but the model…”
• TinyViT: Fast Pretraining Distillation for Small Vision Transformers (2022, methodology): “During training, we only reuse the stored sparse labels for distillation with label smoothing [61,58], which is defined as where ˆ y c is the recovered teacher logits for student model distillation, i.e. , ˆ y = [ˆ y 1 , . . . , ˆ y c , . . . , ˆ y C ].”
• Knowledge Inheritance for Pre-trained Language Models (2021, unknown): “Previous work also indicates the relation between KD and label smoothing (Shen et al., 2021), however, we show in appendix F that the improvements of KI are not because of benefiting from optimizing smoothed targets, which impose regularization.”
• MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks (2020, unknown): “Therefore, the distilled students will encourage representations of examples to lie in tight equally separated clusters and enforce similar instances more distinguishable in feature space, similar to label smoothing [36, 44].”
• Meta Knowledge Distillation (2022, result): “In contrast, Shen et al. (2021) reached a different conclusion showing that teachers trained with label smoothing produce better students.”
• LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS (2023, background): “Knowledge Distillation Knowledge Distillation (KD) is widely adopted in various machine learning tasks [50, 53, 60, 61].”