Empirical Analysis of Knowledge Distillation Technique for Optimization of Quantized Deep Neural Networks

By Sungho Shin, Yoonho Boo, Wonyong Sung. 2019

6 citations recorded by the source corpus.

Knowledge distillation (KD) is a very popular method for model size reduction. Recently, the technique is exploited for quantized deep neural networks (QDNNs) training as a way to restore the performance sacrificed by word-length reduction. KD, however, employs additional hyper-parameters, such as temperature, coefficient, and the size of teacher network for QDNN training. We analyze the effect of these hyper-parameters for QDNN optimization with KD. We find that these hyper-parameters are inter-related, and also introduce a simple and effective technique that reduces \textit{coefficient} during training. With KD employing the proposed hyper-parameters, we achieve the test accuracy of 92.7% and 67.0% on Resnet20 with 2-bit ternary weights for CIFAR-10 and CIFAR-100 data sets, respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

Citations in the corpus

• Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks (2020, unknown): “In the case of QDNNs, the quantized student network can compensate for its accuracy loss via supervision of the teacher model (Mishra and Marr 2018; Polino, Pascanu, and Alistarh 2018; Shin, Boo, and Sung 2019; Kim et al. 2019).”