Towards Understanding Knowledge Distillation

By Mary Phuong, Christoph H. Lampert. 2019

400 citations recorded by the source corpus.

Knowledge distillation, i.e. one classifier being trained on the outputs of another classifier, is an empirically very successful technique for knowledge transfer between classifiers. It has even been observed that classifiers learn much faster and more reliably if trained with the outputs of another classifier as soft labels, instead of from ground truth data. So far, however, there is no satisfactory theoretical explanation of this phenomenon. In this work, we provide the first insights into the working mechanisms of distillation by studying the special case of linear and deep linear classifiers. Specifically, we prove a generalization bound that establishes fast convergence of the expected risk of a distillation-trained linear classifier. From the bound and its proof we extract three keyfactors that determine the success of distillation: data geometry – geometric properties of the datadistribution, in particular class separation, has an immediate influence on the convergence speed of the risk; optimization bias– gradient descentoptimization finds a very favorable minimum of the distillation objective; and strong monotonicity– the expected risk of the student classifier always decreases when the size of the training set grows.

Subfields: Artificial Intelligence, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, background): “From theoretical perspective, [36] provides analysis for linear case.”
Knowledge Distillation: A Survey (2020, methodology): “Despite a huge number of the knowledge distillation methods and applications, the understanding of knowledge distillation including theoretical explanations and empiri-123 cal evaluations remains insufﬁcient (Lopez-Paz et al. 2016; Phuong and Lampert 2019a; Cho and Hariharan 2019).”
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020, methodology): “LUCIR is based on three main components: (1) cosine normalization balances the magnitudes of past and new class probabilities, (2) less-forget constraint modifies the usual distillation loss to handle feature vectors instead of raw scores and (3) inter-class separation encourages the network to sepa”
Self-Distillation Amplifies Regularization in Hilbert Space (2020, background): “Recently [26] analyzed why the student model is able to mimic teacher model in knowledge distillation and [21] presented a statistical perspective on distillation.”
Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications (2020, background): “This logit contains more information than its one-hot prediction, thereby training the student model faster than the standard training with much fewer samples [144].”
Peer Collaborative Learning for Online Knowledge Distillation (2020, background): “Traditionally, knowledge distillation requires to pre-train a high-capacity teacher model in the ﬁrst stage, and then transfer the knowledge of the teacher to a smaller student model in the second stage [6, 18, 17].”
Lifelong Twin Generative Adversarial Networks (2021, background): “Traditional knowledge distribution approaches normally train a classifier on the predictions made by another classifier [17].”
Search to Distill: Pearls Are Everywhere but Not the Eyes (2019, background): “A recent work [23] showed that the data geometry, such as margin between classes, can strongly benefit KD in the case of a single-layer linear network.”
GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation (2021, methodology): “Knowledge Distillation Knowledge distillation (Phuong and Lampert, 2019) is a technique that trains a smaller student classifier on the outputs of a”
Hydra: Preserving Ensemble Diversity for Model Distillation (2020, background): “Also, distillation has been recently the topic of theoretical analysis to better explain its empirical success (Lopez-Paz et al., 2015; Phuong & Lampert, 2019).”
Universal Representation Learning from Multiple Domains for Few-shot Classification (2021, background): “Our work is related to knowledge distillation (KD) methods [17, 27, 30, 40, 45, 50] that dis-”
A statistical perspective on distillation (2021, methodology): “While several works have studied various aspects of distillation (Lopez-Paz et al., 2016; Phuong & Lampert, 2019; Mobahi et al., 2020; Ji & Zhu, 2020; Zhang & Sabuncu, 2020; Zhou et al., 2021; Hsu et al., 2021) (cf. § 2.2), precise answers to these questions remain elusive.”
Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher (2020, methodology): “Our approach and [18] differ from classic learning theory for generalization bound of neural network [17, 4, 1].”
Self-Distillation as Instance-Specific Label Smoothing (2020, background): “[31] considered a special case of distillation using linear and deep linear classifiers, and theoretically analyzed the effect of distillation on student models.”
Prompt-aligned Gradient for Prompt Tuning (2022, methodology): “Motivated by the success of knowledge distillation [34, 16] in knowledge transfer, we leverage the zero-shot CLIP predictions as the general knowledge, and compare the fine-tuned predictions with the general knowledge to regularize the gradient direction.”
Decoupled Knowledge Distillation (2022, background): “Besides, several works also focus on interpreting the classical KD method [2, 26].”