On the Efficacy of Knowledge Distillation

By Jang Hyun Cho, Bharath Hariharan. 2019

788 citations recorded by the source corpus.

In this paper, we present a thorough evaluation of the efficacy of knowledge distillation and its dependence on student and teacher architectures. Starting with the observation that more accurate teachers often don’t make good teachers, we attempt to tease apart the factors that affect knowledge distillation performance. We find crucially that larger models do not often make better teachers. We show that this is a consequence of mismatched capacity, and that small students are unable to mimic large teachers. We find typical ways of circumventing this (such as performing a sequence of knowledge distillation steps) to be ineffective. Finally, we show that this effect can be mitigated by stopping the teacher’s training early. Our results generalize across datasets and models.

Subfields: Artificial Intelligence, Computer Science Applications, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Efficiency & Compression

Citations in the corpus

Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “We build on top of the well-known Knowledge Distillation framework [9, 2, 35], the effectiveness of which has been extensively studied [12, 56].”
Knowledge Distillation: A Survey (2020, background): “Despite a huge number of the knowledge distillation methods and applications, the understanding of knowledge distillation including theoretical explanations and empiri-123 cal evaluations remains insufﬁcient (Lopez-Paz et al. 2016; Phuong and Lampert 2019a; Cho and Hariharan 2019).”
Task-Oriented Feature Distillation (2020, background): “find that the teacher models trained with early-stopping always lead to accuracy boost on student models [10].”
Adaptive Curriculum Learning (2021, background): “KL divergence is frequently used to maintain useful knowledge learned from a pre-trained model [4, 5, 17, 43].”
A statistical perspective on distillation (2021, background): “Prior work has observed that mismatch between the sizes of the student and teacher can also affect distillation (Cho & Hariharan, 2019; Mirzadeh et al., 2020).”
Prime-Aware Adaptive Distillation (2020, background): “Knowledge Distillation (KD) methods [13,28,40,24,32,6] are effective for this purpose and have been extensively explored in the past few years.”
SEED: Self-supervised Distillation For Visual Representation (2021, neutral): “For example in image classification task (Mishra & Marr, 2017; Shen & Savvides, 2020; Polino et al., 2018; Cho & Hariharan, 2019), it is normally a cross entropy loss.”
Curriculum-Based Meta-learning (2021, methodology): “We achieve the goal by using the strategy of Knowledge Distillation (KD) [7].”
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time (2023, methodology): “…al., 2019; Zhao et al., 2019), pruning (Molchanov et al., 2016; Liu et al., 2018; He et al., 2019; Hoefler et al., 2021), and distillation (Hinton et al., 2015; Cho & Hariharan, 2019; Tang et al., 2019; Touvron et al., 2021) have been applied to speed up the inference of the machine learning model.”
ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation (2021, background): “[14] proposes to early stop the teacher to reduce the gap.”
MetaCorrection: Domain-aware Meta Loss Correction for Unsupervised Domain Adaptation in Semantic Segmentation (2021, neutral): “Previous deep supervision approaches [4, 40] directly forces the low-level output to mimic the one-hot pseudo label computed from high-level output layer, which may bring about supervision bias and eliminate useful detailed information in low-level layer due to the capacity gap [8, 25].”
VideoMamba: State Space Model for Efficient Video Understanding (2024, neutral): “For evaluation, we undertake zero-shot video-text retrieval tasks across ﬁve prominent benchmarks, including MSRVTT [87], DiDeMo [1], ActivityNet [31], LSMDC [62], and MSVD [10].”
Meta Knowledge Distillation (2022, methodology): “All the mentioned problems reported by previous methods are based on experiments with a fixed recommended temperature, i.e. 4 in (Tian et al., 2019; Cho & Hariharan, 2019).”
Knowledge Transfer Graph for Deep Collaborative Learning (2019, background): “[6] also pointed out that larger models do not often make better teachers.”
Decoupled Knowledge Distillation (2022, neutral): “What’s more, related works on this problem also could be explained from this perspective, e.g ., ESKD [3] employs early-stopped teacher models to alleviate this problem, and these teachers would be under-convergence and yield smaller p T t .”