Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy

By Asit K. Mishra, Debbie Marr. 2017

354 citations recorded by the source corpus.

Deep learning networks have achieved state-of-the-art accuracies on computer vision workloads like image classification and object detection. The performant systems, however, typically involve big models with numerous parameters. Once trained, a challenging aspect for such top performing models is deployment on resource constrained inference systems - the models (often deep networks or wide networks or both) are compute and memory intensive. Low-precision numerics and model compression using knowledge distillation are popular techniques to lower both the compute requirements and memory footprint of these deployed models. In this paper, we study the combination of these two techniques and show that the performance of low-precision networks can be significantly improved by using knowledge distillation techniques. Our approach, Apprentice, achieves state-of-the-art accuracies using ternary precision and 4-bit precision for variants of ResNet architecture on ImageNet dataset. We present three schemes using which one can apply knowledge distillation techniques to various stages of the train-and-deploy pipeline.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “Some KD methods have been proposed using the quantization process in the teacher–student framework (Polino et al. 2018; Mishra and Marr 2018; Wei et al. 2018; Shin et al. 2019; Kim et al. 2019a).”
Relational Knowledge Distillation (2019, background): “[25] and Mishra and Marr [20] combine KD with network quantization, which aims to reduce bit precision of weights and activations.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Model distillation [3, 94, 148, 173, 189, 200, 260, 262, 280] is a method in which a large model with higher accuracy is used as a teacher to help the training of a compact student model.”
Quantizing deep convolutional networks for efficient inference: A whitepaper (2018, background): “Distilled training to further improve the accuracy of quantized models [32].”
On the Efficacy of Knowledge Distillation (2019, background): “The last of these, knowledge distillation, is a generalpurpose technique that at first glance is widely applicable and complements all other ways of compressing neural networks [20].”
I-BERT: Integer-only BERT Quantization (2021, neutral): “These techniques can be broadly categorized into (1) model pruning [18, 31, 35, 38, 40, 67], (2) knowledge distillation [21, 39, 43, 49, 70], (3) efficient neural architecture design [23, 24, 37, 51, 57], (4) hardware and neural architecture co-design [16, 17, 22, 29, 64], and (5) quantization [5, 7”
Learned Step Size Quantization (2019, methodology): “…PACT (Choi et al., 2018b), Regularization (Choi et al., 2018c), and NICE (Baskin et al., 2018). stochastic gradient descent to learn a quantizer that minimizes task loss (Zhu et al., 2016; Mishra & Marr, 2017; Choi et al., 2018b;a; Jung et al., 2018; Baskin et al., 2018; Polino et al., 2018).”
Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks (2019, background): “[28] introduces knowledge distillation to improve quantized networks’ performance.”
Explicit Loss-Error-Aware Quantization for Low-Bit Deep Neural Networks (2018, methodology): “Even for the latest ternary ResNet-18 model distilled by a much deeper and more powerful ResNet-152 model [26], it still has −1.”
Does Knowledge Distillation Really Work? (2021, background): “Knowledge distillation can improve model efficiency [30, 34], unsupervised domain adaptation [29], improved object detection [9], model transparency [37], and adversarial robustness [14, 33].”
HAWQV3: Dyadic Neural Network Quantization (2020, background): “(iv) Distilling the knowledge from pre-trained model to a compact student model [23, 36, 42, 55].”
SEED: Self-supervised Distillation For Visual Representation (2021, neutral): “For example in image classification task (Mishra & Marr, 2017; Shen & Savvides, 2020; Polino et al., 2018; Cho & Hariharan, 2019), it is normally a cross entropy loss.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “6 [16], [38], [81], [85], [152], [212], [230] [5], [37], [81], [131], [162], [162], [230], [247]”
Knowledge Distillation for Optimization of Quantized Deep Neural Networks (2019, methodology): “Apprentice [19] showed that employing a pretrained teacher model is better than training teacher and student simultaneously from scratch.”
An Empirical study of Binary Neural Networks' Optimisation (2018, neutral): “Mishra & Marr (2018) reported similar results.”
LIT: Learned Intermediate Representation Training for Model Compression (2019, result): “This corroborates prior observations in Mishra & Marr (2018).”
Joint Training of Low-Precision Neural Network with Quantization Interval Parameters (2018, methodology): “Recently, the distillation techniques have been applied to improve performance on training low-precision networks [13; 14].”
Distilled Person Re-Identification: Towards a More Scalable System (2019, methodology): “In most knowledge distillation methods [21, 37, 16], soft labels are utilized.”