Does Knowledge Distillation Really Work?

By Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A. Alemi, Andrew Gordon Wilson. 2021

292 citations recorded by the source corpus.

Knowledge distillation is a popular technique for training a small student network to emulate a larger teacher model, such as an ensemble of networks. We show that while knowledge distillation can improve student generalization, it does not typically work as it is commonly understood: there often remains a surprisingly large discrepancy between the predictive distributions of the teacher and the student, even in cases when the student has the capacity to perfectly match the teacher. We identify difficulties in optimization as a key reason for why the student is unable to match the teacher. We also show how the details of the dataset used for distillation play a role in how closely the student matches the teacher -- and that more closely matching the teacher paradoxically does not always lead to better student generalization.

Subfields: Efficiency & Compression, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

In Evaluation & Benchmarks

Citations in the corpus

Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “Ensemble models on the other hand, have been shown to be promising teachers from the early work of [9] until recent works in various domains [10, 68, 54, 56] and with techniques to boost the their performance [52, 17, 41].”
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023, background): “Stanton et al. (2021) and Furlanello et al. (2018) report a related observation in the context of knowledge distillation: it is surprisingly hard for models to fit the predictions of other models, even when they have sufficient capacity to do so.”
A Fast Knowledge Distillation Framework for Visual Recognition (2021, background): “Recently, many variants and extensions are proposed [6, 18, 24–26, 35, 37, 44, 47, 49, 54], such as employing internal feature representations [30], adversarial training with discriminators [32], transfer ﬂow [48], contrastive distillation [40], patient and consistent [2] etc.”