Knowledge Distillation for Optimization of Quantized Deep Neural Networks
By Sungho Shin, Yoonho Boo, Wonyong Sung. 2019
20 citations recorded by the source corpus.
Knowledge distillation (KD) technique that utilizes a pretrained teacher model for training a student network is exploited for the optimization of quantized deep neural networks (QDNNs). We consider the choice of the teacher network and also investigate the effect of hyperparameters for KD. We employ several large floating-point and quantized models as the teacher network. The experiment shows that the softmax distribution produced by the teacher network is more important than its performance for effective KD training. Since the softmax distribution of the teacher network can be controlled by KD's hyperparameters, we analyze the interrelationship of each KD component for quantized DNN training. Our experiments show that even a small teacher model can achieve the same distillation performance as a large teacher model. We also propose the gradual soft loss reduction (GSLR) technique which controls the mixing ratio of hard and soft losses during training for robust KD based QDNN optimization.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
Citations in the corpus
- Knowledge Distillation: A Survey (2020, methodology): “Some KD methods have been proposed using the quantization process in the teacher–student framework (Polino et al. 2018; Mishra and Marr 2018; Wei et al. 2018; Shin et al. 2019; Kim et al. 2019a).”
- Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks (2020, unknown): “In the case of QDNNs, the quantized student network can compensate for its accuracy loss via supervision of the teacher model (Mishra and Marr 2018; Polino, Pascanu, and Alistarh 2018; Shin, Boo, and Sung 2019; Kim et al. 2019).”