QKD: Quantization-aware Knowledge Distillation

By Jangho Kim, Yash Bhalgat, Jinwon Lee, Chirag Patel, Nojun Kwak. 2019

82 citations recorded by the source corpus.

Quantization and Knowledge distillation (KD) methods are widely used to reduce memory and power consumption of deep neural networks (DNNs), especially for resource-constrained edge devices. Although their combination is quite promising to meet these requirements, it may not work as desired. It is mainly because the regularization effect of KD further diminishes the already reduced representation power of a quantized model. To address this short-coming, we propose Quantization-aware Knowledge Distillation (QKD) wherein quantization and KD are care-fully coordinated in three phases. First, Self-studying (SS) phase fine-tunes a quantized low-precision student network without KD to obtain a good initialization. Second, Co-studying (CS) phase tries to train a teacher to make it more quantizaion-friendly and powerful than a fixed teacher. Finally, Tutoring (TU) phase transfers knowledge from the trained teacher to the student. We extensively evaluate our method on ImageNet and CIFAR-10/100 datasets and show an ablation study on networks with both standard and depthwise-separable convolutions. The proposed QKD outperformed existing state-of-the-art methods (e.g., 1.3% improvement on ResNet-18 with W4A4, 2.6% on MobileNetV2 with W4A4). Additionally, QKD could recover the full-precision accuracy at as low as W3A3 quantization on ResNet and W6A6 quantization on MobilenetV2.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Tijmen Blankevoort
3. Markus Nagel
4. Qun Liu
5. Xin Jiang
6. Lu Hou
7. Itay Hubara
8. Daniel Soudry

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Tijmen Blankevoort
3. Markus Nagel
4. Wei Zhang
5. Qun Liu
6. Lu Hou
7. Daniel Soudry
8. Xin Jiang

Citations in the corpus

• Knowledge Distillation: A Survey (2020, methodology): “Rather than using the ensemble of logits, Kim et al. (2019b) introduced a feature fusion module to construct the teacher classifier.”
• TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020, methodology): “Although combining quantization and distillation has been explored in convolutional neural networks (CNNs) (Polino et al., 2018; Stock et al., 2020; Kim et al., 2019), using knowledge distillation to train quantized BERT has not been studied.”
• Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming (2020, methodology): “1 we compare AdaQuant to current state-ofthe-art methods including QAT with knowledge distillation (QAT-KLD) (Kim et al., 2019) and AdaRound (Nagel et al.”
• Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “Designing effi-cient strategies for training low-precision [71, 29, 70] or any-precision networks [27, 65] that can flexibly adjust the precision during inference is also another recent trend in quantization.”
• KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization (2021, methodology): “(2019), Kim et al. (2019). Although TernaryBERT Zhang et al. (2020) is designed based on KD along with quantization, such a method mainly provides 2-bit and 8-bit weight quantization rather than arbitrary bit quantization.”
• Accurate Post Training Quantization With Small Calibration Sets (2021, methodology): “1 we compare AdaQuant to current state-of-the-art methods including QAT with knowledge distillation (QAT-KLD) (Kim et al., 2019) and AdaRound (Nagel et al., 2020).”
• Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks (2020, unknown): “Recently, several papers have adopted KD to restore the accuracy loss due to the quantization error of reduced-precision inference (Zhuang et al. 2018; Polino, Pascanu, and Alistarh 2018; Mishra and Marr 2018; Shin, Boo, and Sung 2019; Kim et al. 2019).”
• LSQ+: Improving low-bit quantization through learnable offsets and better initialization (2020, background): “In a parallel line of research, some works [15, 18, 21] have tried to apply knowledge distillation to quantization resulting in improved performances.”