Learned Step Size Quantization

By Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, Dharmendra S. Modha. 2019

1,090 citations recorded by the source corpus.

Deep networks run with low precision operations at inference time offer power and space advantages over high precision alternatives, but need to overcome the challenge of maintaining high accuracy as precision decreases. Here, we present a method for training such networks, Learned Step Size Quantization, that achieves the highest accuracy to date on the ImageNet dataset when using models, from a variety of architectures, with weights and activations quantized to 2-, 3- or 4-bits of precision, and that can train 3-bit models that reach full precision baseline accuracy. Our approach builds upon existing methods for learning weights in quantized networks by improving how the quantizer itself is configured. Specifically, we introduce a novel means to estimate and scale the task loss gradient at each weight and activation layer's quantizer step size, such that it can be learned in conjunction with other network parameters. This approach works using different levels of precision as needed for a given system and requires only a simple modification of existing training code.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “For example, using Hessian information (Dong et al., 2019), step-size quantization (Esser et al., 2019), soft quantization (Gong et al., 2019), mixed-precision via linear programming optimization (Yao et al., 2021), and other learned quantization methods (Zhang et al., 2018; Gholami et al., 2021).”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “[29] combines knowledge distillation [28] and learned step size quantization [19] to quantize the weight to 2–8 bits.”
Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, methodology): “LSQ treats the scalar s as a learnable parameter, which is updated by a gradient-based optimization algorithm.”
Differentiable Model Compression via Pseudo Quantization Noise (2021, methodology): “Jain et al. (2019); Esser et al. (2020), and Bhalgat et al. (2020) proposed learning the quantizer step-size or dynamicrange using STE, but do not allow to select the bitdwidth.”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, methodology): “…computer vision models (Lin et al., 2016; Jacob et al., 2018; Krishnamoorthi, 2018; Zhang et al., 2018; Choukroun et al., 2019; Dong et al., 2019; Esser et al., 2019; Nagel et al., 2019, 2020), there is relatively little work done on quantizing NLP models (Wang et al., 2018b; Xu et al., 2018),…”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Notable work here are LQNets [268], PACT [36], LSQ [56], and LSQ+ [15] which jointly optimizes the clipping range and the weights in NN during training.”
BinaryBERT: Pushing the Limit of BERT Quantization (2020, methodology): “Thus we use Learned Step-size Quantization (LSQ) (Esser et al., 2019) to directly learn the quantized values, which empirically achieves better quantization performance.”
Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollers (2019, background): “The works [12, 6] exploits learning-based approaches for determining the quantization ranges of activation and weights at low-bitwidth precision.”
Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models (2022, background): “Among them, quantization [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], working in the low-precision arithmetic fashion, is one of the key approaches to compress large models and fit them into the lightweight devices.”
Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation (2021, methodology): “Compared to uniform quantization, N2UQ allows more flexibility in learning input thresholds, and the results demonstrate that N2UQ surpasses uniform methods [8, 12, 14, 50] by a large margin.”
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, neutral): “Given a stream of audio, IMU, and RGB video data, a model must identify the keystep being performed at each frame, as well as decide which sensor(s) to use for subsequent time-steps.”
A compute-in-memory chip based on resistive random-access memory (2022, methodology): “A similar phenomenon has been reported in neural-network quantization literature where a model trained with quantization occasionally outperforms a full-precision model 54,55 .”
Differentiable Joint Pruning and Quantization for Hardware Efficiency (2020, methodology): “44 – LSQ [7] 3/3 70.”
Q-Diffusion: Quantizing Diffusion Models (2023, methodology): “For activation quantization, since activations keep changing over inputs, doing adaptive rounding is infeasible and we only adjust step sizes of quantizers according to [11].”
Compression of Generative Pre-trained Language Models via Quantization (2022, methodology): “We compare the following representative quantization methods including (i) LAQ (Zhang et al., 2020) for BERT; (ii) PACT (Choi et al., 2018) and LSQ (Esser et al., 2020)) for computer vision tasks, to generative pre-trained model, GPT-2.”
F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (2022, methodology): “We ﬁrst apply our method using conventional training (Choi et al., 2018; Esser et al., 2019; Jin et al., 2020b; Fu et al., 2021), where the quantized model is trained with the simplest setting as those for full-precision model (more details in Appx.”
KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization (2021, methodology): “(2020), Zadeh & Moshovos (2020), product quantization (PQ) Fan et al. (2020), and ternary quantization Zhang et al. (2020). However, different quantization methods include different features: mixed-precision quantization is not always hardware-friendly, PQ needs extra clustering operations during th”