Knowledge Distillation: A Survey
By Jianping Gou, Baosheng Yu, S. Maybank, Dacheng Tao. 2020
4,395 citations recorded by the source corpus.
In recent years, deep neural networks have been successful in both industry and academia, especially for computer vision tasks. The great success of deep learning is mainly due to its scalability to encode large-scale data and to maneuver billions of model parameters. However, it is a challenge to deploy these cumbersome deep models on devices with limited resources, e.g., mobile phones and embedded devices, not only because of the high computational complexity but also the large storage requirements. To this end, a variety of model compression and acceleration techniques have been developed. As a representative type of model compression and acceleration, knowledge distillation effectively learns a small student model from a large teacher model. It has received rapid increasing attention from the community. This paper provides a comprehensive survey of knowledge distillation from the perspectives of knowledge categories, training schemes, teacher–student architecture, distillation algorithms, performance comparison and applications. Furthermore, challenges in knowledge distillation are briefly reviewed and comments on future research are discussed and forwarded.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Dacheng Tao
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Yoshua Bengio
- Quoc V. Le
- Jun Zhu
In Computer Vision and Pattern Recognition
- Dacheng Tao
- Luc Van Gool
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Yi Yang
- Jitendra Malik
- Bernt Schiele
In Efficiency & Compression
Citations in the corpus
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023, methodology): “Model distillation (Hinton et al., 2015; Gou et al., 2021) instead reduces model size at a given precision, using data generated from the larger model to finetune the smaller model.”
- Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, background): “It is common to use a fixed teacher because repeating experiments and selecting the best teacher is expensive [6, 19].”
- Choose Your Weapon: Survival Strategies for Depressed AI Academics (2023, methodology): “Neuroevolution and neural architecture search [10], and knowledge distillation [6], [13] methods are only a few of the available methods for edge AI.”
- A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “This strategy typically uses the previously-learned model as the teacher and the currently-trained model as the student, while implementing knowledge distillation (KD) [117] to mitigate catastrophic forgetting.”
- Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation (2022, methodology): “detection contain logits together with the offset of bounding box [12].”
- Few-Shot Class-Incremental Learning via Relation Knowledge Distillation (2021, methodology): “Knowledge distillation is a common and effective approach to transfer knowledge from an old network to a new one in CIL (Gou et al. 2020).”
- ImageNet-21K Pretraining for the Masses (2021, methodology): “In addition to the general benefit of KD training [16], semantic KD has another significant advantage - it can predict the missing tags that arise from the inconsistent tagging of ImageNet-21K-P.”
- Learning to Ground Visual Objects for Visual Dialog (2021, methodology): “Knowledge Distillation Knowledge distillation (Hinton et al., 2015; Gou et al., 2020; Mirzadeh et al., 2020; Romero et al., 2014; Liu et al., 2019; Jiao et al., 2019) has been widely used in different fields of artificial intelligence, including visual recognition, speech, recognition, natural langu”
- Medical Image Segmentation Review: The Success of U-Net (2022, methodology): “Deep model compression techniques such as pruning [201], quantization [201], lowrank approximation [202], knowledge distillation [203], and neural architecture search [204] to new a few, is an area”
- Large Language Models Are Reasoning Teachers (2022, result): “We note that our work could also be considered a distant variant of KD (see Gou et al. (2021) for a survey), similar to works on improving prompt-based methods such as Yoo et al. (2021); Schick and Schütze (2021b,a); Zelikman et al. (2022).”
- Federated Learning on Non-IID Data: A Survey (2021, background): “Knowledge distillation Knowledge distillation [9, 50, 41] is also a promising idea for personalized federated learning.”
- Language Models Can Teach Themselves to Program Better (2022, methodology): “Each of the 3 GPT-Neo model sizes was fine-tuned for 1 epoch using each of the 4 different 1M length synthetic verified datasets generated, yielding 12 fine-tuning runs with those results shown in Figure 6.”
- A Survey on Model Compression for Large Language Models (2023, positive): “This approach is similar to vanilla KD [Gou et al., 2021; Park et al., 2019; Zhao et al., 2022; Liu et al., 2021a], but with the distinction that the teacher models are LLMs.”
- Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, background): “Knowledge Distillation [74, 196] Hardware Accelerator [141, 192]”
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023, background): “…pseudo-labels is widely used in semi-supervised learning (Laine & Aila, 2016; Tarvainen & Valpola, 2017; Xie et al., 2020), domain adaptation (French et al., 2017; Shu et al., 2018), and knowledge distillation (Hinton et al., 2015; Gou et al., 2021; Stanton et al., 2021; Beyer et al., 2022).”
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (2023, neutral): “In the knowledge distillation domain (Gou et al., 2021), the student model (JudgeLM) mimics the teacher model (GPT-4) to achieve competitive or even superior performance.”