Understanding and Improving Knowledge Distillation

By Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H. Chi, Sagar Jain. 2020

164 citations recorded by the source corpus.

Knowledge distillation is a model-agnostic technique to improve model quality while having a fixed capacity budget. It is a commonly used technique for model compression, where a higher capacity teacher model with better quality is used to train a more compact student model with better inference efficiency. Through distillation, one hopes to benefit from student's compactness, without sacrificing too much on model quality. Despite the large success of knowledge distillation, better understanding of how it benefits student model's training dynamics remains under-explored. In this paper, we dissect the effects of knowledge distillation into three main factors: (1) benefits inherited from label smoothing, (2) example re-weighting based on teacher's confidence on ground-truth, and (3) prior knowledge of optimal output (logit) layer geometry. Using extensive systematic analyses and empirical studies on synthetic and real-world datasets, we confirm that the aforementioned three factors play a major role in knowledge distillation. Furthermore, based on our findings, we propose a simple, yet effective technique to improve knowledge distillation empirically.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Knowledge Distillation: A Survey (2020, background): “Knowledge distillation has also been explored for label smoothing, for assessing the accuracy of the teacher and for obtaining a prior for the optimal output layer geometry (Tang et al. 2020).”
Localization Distillation for Object Detection (2022, background): “Therefore, as a direct corollary of [61], LD holds the gradient rescaling to the distribution focal loss (DFL) [38] w.”
Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher (2020, background): “[21] argues that KD not only transfers super-class correlations to students, but also gives higher credits to correct samples in gradient flow.”
Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks (2020, unknown): “Intuitively, using a good teacher when training QDNN with KD will improve the performance (Tang et al. 2020).”
Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation (2020, background): “While standard distillation confers smoothing benefits (where the teacher may provide higher-quality prediction targets to the student [5, 8]), it incurs a student-teacher statistical approximation-error of similar magnitude as when training directly on original labeled dataset.”
Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation (2021, background): “This result is consistent with prior studies that addressed the efficacy of “soft” targets [Furlanello et al., 2018; Tang et al., 2020].”
Feature Normalized Knowledge Distillation for Image Classification (2020, background): “Moreover, Tang et al. [33] broke down the eﬀects of KD into label smoothing, example re-weighting and prior knowledge of optimal output layer geometry three aspects which is similar to [25].”
Transferring Inductive Biases through Knowledge Distillation (2020, background): “This could be due to the regularization effect of distillation [26, 35].”
Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study (2021, result): “This rising idea is becoming more and more dominant and has been quoted by a large number of recent literatures (Arani et al., 2019; Tang et al., 2020; Mghabbar & Ratnamogan, 2020; Shen et al., 2020; Khosla et al., 2020).”
TinyViT: Fast Pretraining Distillation for Small Vision Transformers (2022, methodology): “Following [63], we can generate the heatmap of Pearson correlation coefficients between classes on the prediction logits.”
A statistical perspective on distillation (2021, background): “This plausibly aids the student by weighting samples differently (Furlanello et al., 2018; Tang et al., 2020).”