Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming

By Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, Daniel Soudry. 2020

157 citations recorded by the source corpus.

Most of the literature on neural network quantization requires some training of the quantized model (fine-tuning). However, this training is not always possible in real-world scenarios, as it requires the full dataset. Lately, post-training quantization methods have gained considerable attention, as they are simple to use and require only a small, unlabeled calibration set. Yet, they usually incur significant accuracy degradation when quantized below 8-bits. This paper seeks to address this problem by introducing two pipelines, advanced and light, where the former involves: (i) minimizing the quantization errors of each layer by optimizing its parameters over the calibration set; (ii) using integer programming to optimally allocate the desired bit-width for each layer while constraining accuracy degradation or model compression; and (iii) tuning the mixed-precision model statistics to correct biases introduced during quantization. While the light pipeline which invokes only (ii) and (iii) obtains surprisingly accurate results; the advanced pipeline yields state-of-the-art accuracy-compression ratios for both vision and text models. For instance, on ResNet50, we obtain less than 1% accuracy degradation while compressing the model to 13% of its original size. We open-sourced our code.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022, neutral): “Alternatively, post-training methods [20, 32, 13, 22], which compress the model in one shot, without retraining, would be very appealing in this setting.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “2) Post-Training Quantization: An alternative to the expensive QAT method is Post-Training Quantization (PTQ) which performs the quantization and the adjustments of the weights, without any ﬁne-tuning [11, 24, 40, 60, 61, 68, 69, 88, 107, 140, 146, 170, 178, 273].”
HAWQV3: Dyadic Neural Network Quantization (2020, background): “The contemporary work of (Hubara et al., 2020) also proposes to use an ILP.”
Post-Training Quantization on Diffusion Models (2022, background): “In order to set quantization parameters for the weight tensor and the activation tensor in a layer, a simple but effective way is to select the quantization parameters that minimize the MSE of the tensors before and after quantization [1, 5, 38, 11].”
F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (2022, methodology): “…a pre-trained full-precision model (He & Cheng, 2018; Nagel et al., 2019; Fang et al., 2020a;b; Garg et al., 2021); Quantization-Aware Training uses training data to optimize quantized models for better performance (Gysel et al., 2018; Esser et al., 2019; Hubara et al., 2020; Tailor et al., 2020).”
BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction (2021, methodology): “, 2020) and the Integer Programming method in AdaQuant (Hubara et al., 2020).”