Knowledge Distillation: A Survey

By Jianping Gou, Baosheng Yu, S. Maybank, Dacheng Tao. 2020

4,395 citations recorded by the source corpus.

In recent years, deep neural networks have been successful in both industry and academia, especially for computer vision tasks. The great success of deep learning is mainly due to its scalability to encode large-scale data and to maneuver billions of model parameters. However, it is a challenge to deploy these cumbersome deep models on devices with limited resources, e.g., mobile phones and embedded devices, not only because of the high computational complexity but also the large storage requirements. To this end, a variety of model compression and acceleration techniques have been developed. As a representative type of model compression and acceleration, knowledge distillation effectively learns a small student model from a large teacher model. It has received rapid increasing attention from the community. This paper provides a comprehensive survey of knowledge distillation from the perspectives of knowledge categories, training schemes, teacher–student architecture, distillation algorithms, performance comparison and applications. Furthermore, challenges in knowledge distillation are briefly reviewed and comments on future research are discussed and forwarded.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Dacheng Tao
  2. Mohit Bansal
  3. Jianfeng Gao
  4. Luke S. Zettlemoyer
  5. Percy Liang
  6. Yoshua Bengio
  7. Quoc V. Le
  8. Jun Zhu

In Computer Vision and Pattern Recognition

  1. Dacheng Tao
  2. Luc Van Gool
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Yi Yang
  7. Jitendra Malik
  8. Bernt Schiele

In Efficiency & Compression

  1. Dan Alistarh
  2. Elias Frantar
  3. Yang He
  4. Dacheng Tao
  5. Lingao Xiao
  6. Zi-Mei Yao
  7. Zhiqiang Shen
  8. Minjia Zhang

Citations in the corpus