LSQ+: Improving low-bit quantization through learnable offsets and better initialization

By Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, Nojun Kwak. 2020

310 citations recorded by the source corpus.

Unlike ReLU, newer activation functions (like Swish, H-swish, Mish) that are frequently employed in popular efficient architectures can also result in negative activation values, with skewed positive and negative ranges. Typical learnable quantization schemes [5], [7] assume unsigned quantization for activations and quantize all negative activations to zero which leads to significant loss in performance. Naively using signed quantization to accommodate these negative values requires an extra sign bit which is expensive for low-bit (2-, 3-, 4-bit) quantization. To solve this problem, we propose LSQ+, a natural extension of LSQ [7], wherein we introduce a general asymmetric quantization scheme with trainable scale and offset parameters that can learn to accommodate the negative activations. Gradient-based learnable quantization schemes also commonly suffer from high instability or variance in the final training performance, hence requiring a great deal of hyper-parameter tuning to reach a satisfactory performance. LSQ+ alleviates this problem by using an MSE-based initialization scheme for the quantization parameters. We show that this initialization leads to significantly lower variance in final performance across multiple training runs. Overall, LSQ+ shows state-of-the-art results for EfficientNet and MixNet and also significantly outperforms LSQ for low-bit quantization of neural nets with Swish activations (e.g.: 1.8% gain with W4A4 quantization and upto 5.6% gain with W2A2 quantization of EfficientNet-B0 on ImageNet dataset). To the best of our knowledge, ours is the first work to quantize such architectures to extremely low bit-widths.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Differentiable Model Compression via Pseudo Quantization Noise (2021, methodology): “(2020), and Bhalgat et al. (2020) proposed learning the quantizer step-size or dynamic-range using STE, but do not allow to select the bitdwidth. Our experiments show that DiffQ outperforms (Esser et al., 2020) (LSQ) both on vision and natural language tasks. Uhlich et al. (2020) proposed a re-param”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, methodology): “We follow a typical setup with symmetric weight and asymmetric activation quantization (Bhalgat et al., 2020).”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Notable work here are LQNets [268], PACT [36], LSQ [56], and LSQ+ [15] which jointly optimizes the clipping range and the weights in NN during training.”
Overcoming Oscillations in Quantization-Aware Training (2022, methodology): “…has been widely adopted in recent literature closing the gap between quantized and full-precision accuracy for a wide range of tasks and networks (Gupta et al., 2015; Zhou et al., 2016; Krishnamoorthi, 2018; Jain et al., 2019; Esser et al., 2020; Bhalgat et al., 2020; Nagel et al., 2021).”
Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models (2022, methodology): “Among them, quantization [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], working in the low-precision arithmetic fashion, is one of the key approaches to compress large models and fit them into the lightweight devices.”
F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (2022, background): “…quantization where the model shares the same precision (Zhou et al., 2018; Wang et al., 2018; Choukroun et al., 2019; Gong et al., 2019; Langroudi et al., 2019; Jin et al., 2020a; Bhalgat et al., 2020; Chen et al., 2020; Yang et al., 2020; Darvish Rouhani et al., 2020; Oh et al., 2021).”
Q-Diffusion: Quantizing Diffusion Models (2023, background): “Therefore, we find the training-free post-training quantization (PTQ) [6, 2, 23, 14, 26] the most applicable for diffusion model compression.”
A White Paper on Neural Network Quantization (2021, methodology): “In later work (Esser et al., 2020; Jain et al., 2019; Bhalgat et al., 2020), the STE is used to calculate the gradient w.”
HAWQV3: Dyadic Neural Network Quantization (2020, methodology): “A common solution is to compress NN models with quantization (Asanovic & Morgan, 1991; Bhalgat et al., 2020; Chin et al., 2020; Dong et al., 2019; Hubara et al., 2016; Jacob et al., 2018; Kim & Kim, 2021; Park et al., 2018a; Rastegari et al., 2016; Sharma et al., 2018; Song et al., 2020; Zhang et…”