Model compression via distillation and quantization

By Antonio Polino, Razvan Pascanu, Dan Alistarh. 2018

843 citations recorded by the source corpus.

Deep neural networks (DNNs) continue to make significant advances, solving tasks from image classification to translation or reinforcement learning. One aspect of the field receiving considerable attention is efficiently executing deep models in resource-constrained environments, such as mobile or embedded devices. This paper focuses on this problem, and proposes two new compression methods, which jointly leverage weight quantization and distillation of larger teacher networks into smaller student networks. The first method we propose is called quantized distillation and leverages distillation during the training process, by incorporating distillation loss, expressed with respect to the teacher, into the training of a student network whose weights are quantized to a limited set of levels. The second method, differentiable quantization, optimizes the location of quantization points through stochastic gradient descent, to better fit the behavior of the teacher model. We validate both methods through experiments on convolutional and recurrent architectures. We show that quantized shallow students can reach similar accuracy levels to full-precision teacher models, while providing order of magnitude compression, and inference speedup that is linear in the depth reduction. In sum, our results enable DNNs for resource-constrained environments to leverage architecture and accuracy advances developed on more powerful devices.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Similarity-Preserving Knowledge Distillation (2019, background): “We believe that similarity-preserving distillation can provide a simple yet effective drop-in replacement for (or complement to) traditional forms of distillation in a variety of application areas, including model compression [28], privileged learning [21], adversarial defense [25], and learning wit”
AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, positive): “Popular techniques include knowledge distillation (Polino et al., 2018), pruning (Han et al.”
Designing Interpretable Approximations to Deep Reinforcement Learning with Soft Decision Trees (2020, neutral): “Briefly, this is related to, but distinct from model compression, which seeks to quantize, code or otherwise process reference network weights in a way that leads to a reduced complexity model of the same structure (Polino, Pascanu, and Alistarh 2018).”
Knowledge Distillation: A Survey (2020, methodology): “For example, quantized knowledge distillation, which can be seen as a parameter pruning method, integrates network quantization into the teacher-student architectures (Polino et al., 2018; Mishra and Marr, 2018; Wei et al., 2018).”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, background): “Knowledge distillation and its improved variants [1, 53, 57, 67, 73] enable teaching students with goals such as quantization [42, 55], compact neural network architecture design [57], semantic segmentation [31], self-distillation [14], and un-/semi-supervised learning [34, 54, 70].”
Lifelong GAN: Continual Learning for Conditional Image Generation (2019, background): “Relevant applications include network compression [26] and network training acceleration [33].”
Differentiable Model Compression via Pseudo Quantization Noise (2021, neutral): “We then compare to the differentiable quantization method by (Polino et al., 2018), which only optimizes the non uniform quantization points, not the pre-quantization weights.”
Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision (2023, methodology): “We believe this can be addressed separately by model distillation and quantization (Polino et al., 2018; Fan et al., 2020).”
Relational Knowledge Distillation (2019, methodology): “In general, conventional KD methods [1, 2, 8, 11, 12, 25, 27, 45, 47] can commonly be expressed as minimizing the objective function:”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “However, the combination of knowledge distillation with prior methods (i.e., quantization and pruning) has shown great success [189]. e) Quantization : Finally, quantization is an approach that has shown great and consistent success in both training and inference of NN models.”
TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020, methodology): “Although combining quantization and distillation has been explored in convolutional neural networks (CNNs) (Polino et al., 2018; Stock et al., 2020; Kim et al., 2019), using knowledge distillation to train quantized BERT has not been studied.”
Online Knowledge Distillation with Diverse Peers (2019, background): “…effective solution for compressing a pre-trained large teacher model into a smaller student model by steering the student predictions towards teacher predictions (Bucilua, Caruana, and Niculescu-Mizil 2006; Ba and Caruana 2014; Hinton, Vinyals, and Dean 2015; Polino, Pascanu, and Alistarh 2018).”
Quantizing deep convolutional networks for efficient inference: A whitepaper (2018, background): “Improved accuracy can be obtained by adapting the quantizer parameters to each kernel within the tensor [17].”
I-BERT: Integer-only BERT Quantization (2021, neutral): “These techniques can be broadly categorized into (1) model pruning [18, 31, 35, 38, 40, 67], (2) knowledge distillation [21, 39, 43, 49, 70], (3) efficient neural architecture design [23, 24, 37, 51, 57], (4) hardware and neural architecture co-design [16, 17, 22, 29, 64], and (5) quantization [5, 7”
Learned Step Size Quantization (2019, background): “To date, most approaches for training low precision networks have employed uniform quantizers, which can be conﬁgured by a single step size parameter (the width of a quantization bin), though more complex nonuniform mappings have been considered (Polino et al., 2018).”
Model Compression with Multi-Task Knowledge Distillation for Web-scale Question Answering System (2019, positive): “Polino et al. (2018) proposed a quantized distillation method.”
Joint Training of Low-Precision Neural Network with Quantization Interval Parameters (2018, methodology): “Recently, the distillation techniques have been applied to improve performance on training low-precision networks [13; 14].”