Loss aware post-training quantization

By Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, A. Bronstein, A. Mendelson. 2019

206 citations recorded by the source corpus.

Neural network quantization enables the deployment of large models on resource-constrained devices. Current post-training quantization methods fall short in terms of accuracy for INT4 (or lower) but provide reasonable accuracy for INT8 (or above). In this work, we study the effect of quantization on the structure of the loss landscape. We show that the structure is flat and separable for mild quantization, enabling straightforward post-training quantization methods to achieve good results. We show that with more aggressive quantization, the loss landscape becomes highly non-separable with steep curvature, making the selection of quantization parameters more challenging. Armed with this understanding, we design a method that quantizes the layer parameters jointly, enabling significant accuracy improvement over current post-training quantization methods. Reference implementation is available at https://github.com/ynahshan/nn-quantization-pytorch/tree/master/lapq.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tijmen Blankevoort
2. Qun Liu
3. Yelysei Bondarenko
4. Daniel Soudry
5. Xin Jiang
6. Lu Hou
7. Itay Hubara
8. Wei Zhang

In Computer Vision and Pattern Recognition

1. Wei Zhang
2. Tijmen Blankevoort
3. Irwin King
4. Dacheng Tao
5. Daniel Soudry
6. Qun Liu
7. Itay Hubara
8. Lu Hou

In Efficiency & Compression

1. Tijmen Blankevoort
2. Lu Hou
3. Yelysei Bondarenko
4. Ron Banner
5. Wei Zhang
6. Daniel Soudry
7. Itay Hubara
8. Qun Liu

Citations in the corpus

• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022, neutral): “Alternatively, post-training methods [20, 32, 13, 22], which compress the model in one shot, without retraining, would be very appealing in this setting.”
• BinaryBERT: Pushing the Limit of BERT Quantization (2020, methodology): “Following (Nahshan et al., 2019), we extract parameters w x , w y from the value layers 2 of multi-head attention in the first two Transformer layers, and assign the following perturbations on parameters: where x ∈ {± 0 .”
• Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (2022, unknown): “The quantization grids for OBQ as well as AdaRound are determined with the same LAPQ [34] procedure that is used by BRECQ.”
• Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models (2022, background): “Among them, quantization [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], working in the low-precision arithmetic fashion, is one of the key approaches to compress large models and fit them into the lightweight devices.”
• Post-Training Quantization on Diffusion Models (2022, unknown): “Based on the aforementioned observations, we establish a baseline of PTQ calibration for DM based on [22], in which the quantized diffusion models are calibrated with samples at time-step t, i.”
• Overcoming Oscillations in Quantization-Aware Training (2022, methodology): “By simulating the quantization operation during training or fine-tuning, the network can adapt to the quantization noise and reach better solutions than with PTQ.”
• Accurate Post Training Quantization With Small Calibration Sets (2021, background): “A per-tensor optimization for reducing the quantization error have been suggested (Nahshan et al., 2019; Banner et al., 2018; Jacob et al., 2018).”