Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation

By Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, Paulius Micikevicius. 2020

488 citations recorded by the source corpus.

Quantization techniques can reduce the size of Deep Neural Networks and improve inference latency and throughput by taking advantage of high throughput integer instructions. In this paper we review the mathematical aspects of quantization parameters and evaluate their choices on a wide range of neural network models for different application domains, including vision, speech, and language. We focus on quantization techniques that are amenable to acceleration by processors with high-throughput integer math pipelines. We also present a workflow for 8-bit quantization that is able to maintain accuracy within 1% of the floating-point baseline on all networks studied, including models that are more difficult to quantize, such as MobileNets and BERT-large.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

The case for 4-bit precision: k-bit Inference Scaling Laws (2022, methodology): “For example, there are methods associated with blocking and grouping (Park et al., 2022; Wu et al., 2020; Jain et al., 2020; Nagel et al., 2019; Krishnamoorthi, 2018; Rusci et al., 2020), centering (Krishnamoorthi, 2018; Jacob et al., 2017), learned data types that are found through clustering…”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, positive): “For matrix multiplication, this was used by Wu et al. (2020) for BERT-sized transformers (350M parameters), while we are the ﬁrst to study vector-wise quantization for autoregressive and large-scale models.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Symmetric quantization is widely adopted in practice for quantizing weights because zeroing out the zero point can lead to reduction in computational cost during inference [247], and also makes the implementation more straightforward.”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, unknown): “Tensor RT [222, 233] is an nVidia developed C++ library that facilitates high-performance inference onNVIDIA graphics processing units (GPUs).”
Knowledge distillation: A good teacher is patient and consistent (2021, background): “Weights quantization [17, 25, 31, 45] and decomposition [5, 9, 23, 43] aim to accelerate and reduce the memory footprint of CNNs by replacing large matrices operations with their lightweight approximations.”
Pareto-Optimal Quantized ResNet Is Mostly 4-bit (2021, methodology): “The authors in [32] study a multitude of pre-trained neural networks, and observe that they can all be quantized to int8 with their accuracies remaining within 1% of the baseline model.”
FP8 Formats for Deep Learning (2022, unknown): “However, if instead we calibrate the tensors to have their own scaling factors (following the convention of int8 quantization to use per-channel and per-tensor scaling factors for weights and activations, respectively [23]) we achieve 10.”
Accelerating Sparse Deep Neural Networks (2021, methodology): “After fine-tuning, the networks are quantized to INT8 using max calibration [41].”
Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale (2021, background): “Most of the many academic works on quantization focus on computer vision [25, 31, 33, 38, 46, 53, 57] or natural language processing models [15, 57, 61, 62].”
F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (2022, methodology): “…eliminate ﬂoating-point operations from the quantized model, integer-only quantization techniques emulate the full-precision multiplication by 32-bit integer multiplication followed by bit shifting (Jacob et al., 2018; Zhu et al., 2020; Wu et al., 2020; Yao et al., 2021; Kim et al., 2021).”
Differentiable Joint Pruning and Quantization for Hardware Efficiency (2020, methodology): “d by deﬁning a continuous relaxation of it and using straight-through estimator for the gradients. We employ a similar technique in this paper. For a general overview of quantization, please refer to [31][15]. Structured pruning approaches can also be summarized into two categories: the onewith ﬁxed”