Revisit Knowledge Distillation: a Teacher-free Framework

By Li Yuan, Francis E. H. Tay, Guilin Li, Tao Wang, Jiashi Feng. 2019

98 citations recorded by the source corpus.

Knowledge Distillation (KD) aims to distill the knowledge of a cumbersome teacher model into a lightweight student model. Its success is generally attributed to the privileged information on similarities among categories provided by the teacher model, and in this sense, only strong teacher models are deployed to teach weaker students in practice. In this work, we challenge this common belief by following experimental observations: 1) beyond the acknowledgment that the teacher can improve the student, the student can also enhance the teacher significantly by reversing the KD procedure; 2) a poorly-trained teacher with much lower accuracy than the student can still improve the latter significantly. To explain these observations, we provide a theoretical analysis of the relationships between KD and label smoothing regularization. We prove that 1) KD is a type of learned label smoothing regularization and 2) label smoothing regularization provides a virtual teacher model for KD. From these results, we argue that the success of KD is not fully due to the similarity information between categories, but also to the regularization of soft targets, which is equally or even more important. Based on these analyses, we further propose a novel Teacher-free Knowledge Distillation (Tf-KD) framework, where a student model learns from itself or manually-designed regularization distribution. The Tf-KD achieves comparable performance with normal KD from a superior teacher, which is well applied when teacher model is unavailable. Meanwhile, Tf-KD is generic and can be directly deployed for training deep neural networks. Without any extra computation cost, Tf-KD achieves up to 0.65\% improvement on ImageNet over well-established baseline models, which is superior to label smoothing regularization. The codes are in: \url{this https URL}

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “, 2018b), self-learning (Yuan et al., 2019), assistant teaching (Mirzadeh et al.”
Self-Distillation as Instance-Specific Label Smoothing (2020, background): “We note that, while previous works have highlighted the similarity between distillation and label smoothing from another perspective [42], we provide a detailed empirical analysis that uncovers additional benefits of instance-specific regularization.”
A Simple Baseline for Pose Tracking in Videos of Crowed Scenes (2020, methodology): “Meanwhile, we apply KD regularization [26] in the class’s logits of the detection model, which can consistently improve the detection results by 0.”
Prime-Aware Adaptive Distillation (2020, background): “Results of baseline and compared methods by our implementation are reliable, as they are almost the same as those of [39], and are higher than those of [32].”
Feature Normalized Knowledge Distillation for Image Classification (2020, methodology): “The most relevant work to ours is Yuan et al. [38], who also realized the relation ship between KD and LSD, however they did not further reveal the underlying causes of both methods.”
Understanding and Improving Knowledge Distillation (2020, background): “For example, Yuan et al. (2019) connects label smoothing to KD.”
Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study (2021, background): “This may challenge the common practice in knowledge distillation (Hinton et al., 2015; Romero et al., 2015) that adopted both hard and soft labels with the cross-entropy loss for distillation, while our surprisingly good results and previous studies (Shen & Savvides, 2020; Shen et al., 2019; Bagheri”
Transferring Inductive Biases through Knowledge Distillation (2020, background): “On the other hand, KD has a regularization effect similar to label smoothing [41, 36].”
Toward Accurate Person-level Action Recognition in Videos of Crowed Scenes (2020, methodology): “Meanwhile, we apply KD regularization [30] in the class’s logits of the detection model, which can consistently improve the detection results by 0.”