Knowledge Distillation Beyond Model Compression

By Fahad Sarfraz, Elahe Arani, Bahram Zonooz. 2020

46 citations recorded by the source corpus.

Knowledge distillation (KD) is commonly deemed as an effective model compression technique in which a compact model (student) is trained under the supervision of a larger pretrained model or an ensemble of models (teacher). Various techniques have been proposed since the original formulation, which mimic different aspects of the teacher such as the representation space, decision boundary, or intra-data relationship. Some methods replace the one-way knowledge distillation from a static teacher with collaborative learning between a cohort of students. Despite the recent advances, a clear understanding of where knowledge resides in a deep neural network and an optimal method for capturing knowledge from teacher and transferring it to student remains an open question. In this study, we provide an extensive study on nine different KD methods which covers a broad spectrum of approaches to capture and transfer knowledge. We demonstrate the versatility of the KD framework on different datasets and network architectures under varying capacity gaps between the teacher and student. The study provides intuition for the effects of mimicking different aspects of the teacher and derives insights from the performance of the different distillation approaches to guide the design of more effective KD methods. Furthermore, our study shows the effectiveness of the KD framework in learning efficiently under varying severity levels of label noise and class imbalance, consistently providing generalization gains over standard training. We emphasize that the efficacy of KD goes much beyond a model compression technique and it should be considered as a general-purpose training paradigm which offers more robustness to common challenges in the real-world datasets compared to the standard training procedure.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bahram Zonooz
2. Elahe Arani
3. Fahad Sarfraz
4. Prashant Bhat

Citations in the corpus

• Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System (2022, positive): “Dark Experience Replay (DER++) applies knowledge distillation (Sarfraz et al., 2021) and regularization on logits sampled during the optimization trajectory.”
• Noise as a Resource for Learning in Knowledge Distillation (2020, background): “While different ways of distilling knowledge to the student have been extensively studied [47], the role of noise in the knowledge distillation framework is not well studied.”