Quantizing deep convolutional networks for efficient inference: A whitepaper

By Raghuraman Krishnamoorthi. 2018

1,281 citations recorded by the source corpus.

We present an overview of techniques for quantizing convolutional neural networks for inference with integer weights and activations. Per-channel quantization of weights and per-layer quantization of activations to 8-bits of precision post-training produces classification accuracies within 2% of floating point networks for a wide variety of CNN architectures. Model sizes can be reduced by a factor of 4 by quantizing weights to 8-bits, even when 8-bit arithmetic is not supported. This can be achieved with simple, post training quantization of weights.We benchmark latencies of quantized networks on CPUs and DSPs and observe a speedup of 2x-3x for quantized implementations compared to floating point on CPUs. Speedups of up to 10x are observed on specialized processors with fixed point SIMD capabilities, like the Qualcomm QDSPs with HVX. Quantization-aware training can provide further improvements, reducing the gap to floating point to 1% at 8-bit precision. Quantization-aware training also allows for reducing the precision of weights to four bits with accuracy losses ranging from 2% to 10%, with higher accuracy drop for smaller networks.We introduce tools in TensorFlow and TensorFlowLite for quantizing convolutional networks and review best practices for quantization-aware training to obtain high accuracy with quantized weights and activations. We recommend that per-channel quantization of weights and per-layer quantization of activations be the preferred quantization scheme for hardware acceleration and kernel optimization. We also propose that future processors and hardware accelerators for optimized inference support precisions of 4, 8 and 16 bits.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

The case for 4-bit precision: k-bit Inference Scaling Laws (2022, methodology): “…and grouping (Park et al., 2022; Wu et al., 2020; Jain et al., 2020; Nagel et al., 2019; Krishnamoorthi, 2018; Rusci et al., 2020), centering (Krishnamoorthi, 2018; Jacob et al., 2017), learned data types that are found through clustering (Gong et al., 2014; Han et al., 2015; Choi et al.,…”
Secure Evaluation of Quantized Neural Networks (2019, methodology): “The goal of this section is to provide an overview of the quantization technique of Jacob et al. [37] (see also [42]) that we will be relying on to get efﬁcient secure inference.”
Differentiable Model Compression via Pseudo Quantization Noise (2021, methodology): “Note that variants of this scheme exist, for instance, symmetric uniform quantization, which enforces that 0 is always exactly represented (Krishnamoorthi 2018).”
Searching for MobileNetV3 (2019, background): “Quantization [23, 25, 47, 41, 51, 52, 37] is another important complementary effort to improve the network efficiency through reduced precision arithmetic.”
Training with Quantization Noise for Extreme Model Compression (2020, methodology): “Closer to our work, several works have focused at simulatenously training and quantizing a network [1, 40, 41, 42].”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, methodology): “running min-max uses exponential moving average of the min and max over multiple batches (Krishnamoorthi, 2018);”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “This scaling factor essentially divides a given range of real values r into a number of partitions (as discussed in [112, 131]): where [ α, β ] denotes the clipping range, a bounded range that we are clipping the real values with, and b is the quantization bit width.”
I-BERT: Integer-only BERT Quantization (2021, methodology): “One promising method to tackle this challenge is quantization [15, 27, 28, 65, 66, 72], a procedure which compresses models into smaller size by representing parameters and/or activations with low bit precision, e.”
The Knowledge Within: Methods for Data-Free Model Compression (2019, methodology): “Throughout our experiments the quantization scheme is fixed to a widely used method described in [15].”
Up or Down? Adaptive Rounding for Post-Training Quantization (2020, methodology): “One effective way to optimize neural networks for inference is neural network quantization (Krishnamoorthi, 2018; Guo, 2018).”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “This allows four multiplies to be dispatched in a three stage pipeline [123], Oq is the quantized output.”
CoDeNet: Efficient Deployment of Input-Adaptive Object Detection on Embedded FPGAs (2021, methodology): “Many prior works have attempted to address this accuracy drop with various techniques, such as non-uniform learnable quantizer [39], mixedprecision quantization [7], progressive fine-tuning [42] as well as group-wise [34] and channel-wise quantization [16].”
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (2023, neutral): “(Jacob et al., 2018; Krishnamoorthi, 2018), Learned step-size quantization (Esser et al., 2019), PACT (Choi et al., 2018), N2UQ (Liu et al., 2022a) and etc, have primarily been developed for convolutional neural networks.”
Ultra-Low Precision 4-bit Training of Deep Neural Networks (2020, background): “4-bit 4-bit inference [21] (where weights and activations are quantized in 4-bit integer formats) has been shown to fully preserve model accuracy and provide signiﬁcant acceleration in comparison to 8-bit integer systems [28] used widely for inference [17].”
Data-Free Quantization Through Weight Equalization and Bias Correction (2019, methodology): “Similar to [18], we observe that the model performance is close to random when quantizing the original model to INT8.”
Accurate Post Training Quantization With Small Calibration Sets (2021, background): “However, naively quantizing a full-precision model to INT4 or lower to accelerate computation usually incurs signiﬁcant accuracy degradation (Krishnamoorthi, 2018; Jacob et al., 2018).”