Low-bit Quantization of Neural Networks for Efficient Inference

By Yoni Choukroun, Eli Kravchik, Pavel Kisilev. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), 2019

463 citations recorded by the source corpus.

Recent machine learning methods use increasingly large deep neural networks to achieve state of the art results in various tasks. The gains in performance come at the cost of a substantial increase in computation and storage requirements. This makes real-time implementations on limited resources hardware a challenging task. One popular approach to address this challenge is to perform low-bit precision computations via neural network quantization. However, aggressive quantization generally entails a severe penalty in terms of accuracy, and often requires retraining of the network, or resorting to higher bit precision quantization. In this paper, we formalize the linear quantization task as a Minimum Mean Squared Error (MMSE) problem for both weights and activations, allowing low-bit precision inference without the need for full network retraining. We propose the analysis and the optimization of constrained MSE problems for performant hardware aware quantization. The proposed approach allows 4 bits integer (INT4) quantization for deployment of pretrained models on limited hardware resources. Multiple experiments on various network architectures show that the suggested method yields state of the art results with minimal loss of tasks accuracy.

Subfields: Computational Mathematics, Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/47a1edfb88f5b4a7ba1e9f6aed327f67f942f6d6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mathematics

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models (2022, methodology): “The former investigation of accuracy impact suggests us taking the final performance into account to find a superior clipping range, where some local optimization methods like [28] are not suitable here.”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, methodology): “…2018b; Zhu et al., 2020); running min-max uses exponential moving average of the min and max over multiple batches (Krishnamoorthi, 2018); MSE ﬁnds quantization parameters that minimize mean squared error between quantized and ﬂoating-point tensors (Choukroun et al., 2019; Banner et al., 2018).”
Post training 4-bit quantization of convolutional networks for rapid-deployment (2018, background): “Their importance can be seen from the recent industrial publications, focusing on quantization methods that avoid re-training (Goncharenko et al., 2018; Choukroun et al., 2019; Meller et al., 2019; Migacz, 2017).”
Post-Training Quantization for Vision Transformer (2021, methodology): “[8] propose the OMSE method to optimize the L2 distance between the quantized tensor and the original tensor.”
ZeroQ: A Novel Zero Shot Quantization Framework (2020, methodology): “82% higher than OMSE [19] with access to training data and 5.”
Post-Training Quantization on Diffusion Models (2022, unknown): “In order to set quantization parameters for the weight tensor and the activation tensor in a layer, a simple but effective way is to select the quantization parameters that minimize the MSE of the tensors before and after quantization [1, 5, 12, 42].”
Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollers (2019, background): “Recently, to ﬁt stringent memory requirements, more aggressive sub-byte precision quantization approaches, i.e. less than 8 bit, are under investigation [3, 12, 6, 13, 16].”
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (2022, unknown): “We begin our experiments with unstructured sparsity, comparing against global magnitude pruning (GMP) [46], the approximate layer-wise OBS method L-OBS [6], and the post-training pruning state-of-the-art method AdaPrune [18].”
LSQ+: Improving low-bit quantization through learnable offsets and better initialization (2020, methodology): “Post-training quantization methods [2, 31, 6] optimize neural networks for quantization without full training and using a little amount of data.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, unknown): “Multiple different metrics have been proposed to ﬁnd the best range, including minimizing Mean Squared Error (MSE) between original unquantized weight distribution and the corresponding quantized values [40, 214, 221, 273].”
Loss aware post-training quantization (2019, methodology): “…quantization perform finer parameter assignment, such as group-wise (Mellempudi et al. 2017), channel-wise (Banner et al. 2018), pixel-wise (Fraser et al. 2018), or filter-wise (Kravchik et al. 2019) quantization, requiring special hardware support and additional computational resources.”
F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (2022, methodology): “One area uses uniform-precision quantization where the model shares the same precision (Zhou et al., 2018; Wang et al., 2018; Choukroun et al., 2019; Gong et al., 2019; Langroudi et al., 2019; Jin et al., 2020a; Bhalgat et al., 2020; Chen et al., 2020; Yang et al., 2020; Darvish Rouhani et al.,…”
Up or Down? Adaptive Rounding for Post-Training Quantization (2020, background): “The same holds for e.g., setting per-layer activation ranges as in (Choukroun et al., 2019).”
A Statistical Framework for Low-bitwidth Training of Deep Neural Networks (2020, background): “Among inference quantization approaches, post training quantization usually does not require access to the partial/full training dataset, and it does not need to re-train/fine-tune the quantized model [1, 2, 3, 4, 5].”