Revisiting Knowledge Distillation via Label Smoothing Regularization

By Li Yuan, Francis E. H. Tay, Guilin Li, Tao Wang, Jiashi Feng. 2020

652 citations recorded by the source corpus.

Knowledge Distillation (KD) aims to distill the knowledge of a cumbersome teacher model into a lightweight student model. Its success is generally attributed to the privileged information on similarities among categories provided by the teacher model, and in this sense, only strong teacher models are deployed to teach weaker students in practice. In this work, we challenge this common belief by following experimental observations: 1) beyond the acknowledgment that the teacher can improve the student, the student can also enhance the teacher significantly by reversing the KD procedure; 2) a poorly-trained teacher with much lower accuracy than the student can still improve the latter significantly. To explain these observations, we provide a theoretical analysis of the relationships between KD and label smoothing regularization. We prove that 1) KD is a type of learned label smoothing regularization and 2) label smoothing regularization provides a virtual teacher model for KD. From these results, we argue that the success of KD is not fully due to the similarity information between categories from teachers, but also to the regularization of soft targets, which is equally or even more important. Based on these analyses, we further propose a novel Teacher-free Knowledge Distillation (Tf-KD) framework, where a student model learns from itself or manually-designed regularization distribution. The Tf-KD achieves comparable performance with normal KD from a superior teacher, which is well applied when a stronger teacher model is unavailable. Meanwhile, Tf-KD is generic and can be directly deployed for training deep neural networks. Without any extra computation cost, Tf-KD achieves up to 0.65\% improvement on ImageNet over well-established baseline models, which is superior to label smoothing regularization.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, background): “Label smoothing, an effective regularizer and related to KD, is particularly related to our work when interpreted as augmenting the output space [69, 53].”
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet (2021, methodology): “A recent concurrent work DeiT [36] applies Knowledge Distillation [16, 49] to improve the original ViT by adding a KD token along with the class token, which is orthogonal to our work, as our T2T-ViT focuses on the architecture design, and our T2T-ViT can achieve higher performance than DeiT without”
MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition (2022, background): “Tf-KD [46] connects label smoothing [32] to Self-KD and applies a manually designed soft label to replace the teacher.”
Channel-wise Knowledge Distillation for Dense Prediction* (2020, background): “Most works on knowledge distillation focus on classification tasks [10, 11, 15, 26, 36, 38, 44].”
All Tokens Matter: Token Labeling for Training Better Vision Transformers (2021, background): “Later, DeiT [36] manages to tackle the data-inefficiency problem by simply adjusting the network architecture and adding an additional token along with the class token for Knowledge Distillation [21, 47] to improve model performance.”
Improving Plasticity in Online Continual Learning via Collaborative Learning (2023, methodology): “The main motivation of DC originates from the idea of entropy regularization-based optimization strategies, like label smoothing [44], knowledge distillation [49], and confidence penalty [36], where we find that overconfidence will hurt performance in non-continual scenarios.”
Auto-scaling Vision Transformers without Training (2022, methodology): “DeiT (Touvron et al., 2020) proposed Knowledge Distillation (KD) (Hinton et al., 2015; Yuan et al., 2020) via a special KD token to improve both performance and training efﬁciency.”
General Instance Distillation for Object Detection (2021, neutral): “[36] proposes that the performance gain from the knowledge distillation mainly due to the regularization of the respond-based knowledge from the teacher model.”
A statistical perspective on distillation (2021, methodology): “Dis-tillation may also be seen as a data-dependent regulariser on the student model (Dong et al., 2019; Yuan et al., 2020”
Knowledge Inheritance for Pre-trained Language Models (2021, neutral): “Out-side NLP, researchers recently demonstrate that a student model could also beneﬁt from a poor teacher for a speciﬁc downstream task (Yuan et al., 2020) (reversed KD).”
Delving Deep Into Label Smoothing (2020, positive): “Our method beneﬁts from the similarities between classes, so it can perform better than Tf-KD [57].”
Refiner: Refining Self-attention for Vision Transformers (2021, methodology): “For a fair comparison with other methods, we ﬁrst replace the SA module with reﬁner on ViT-Base [19] model as it is the most frequently used one [19, 48, 61, 49].”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, background): “Various applications with impressive results have been developed for KD, such as mutual learning [190], teacher assistant [191], and self-learning [192].”
Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, methodology): “Since the self-learning via self-distillation can improve the power of knowledge transfer [23], CTSL-MKT further introduces the self-learning of each peer network into collaborative learning via the response-based knowledge self-distillation.”