A Survey of Quantization Methods for Efficient Neural Network Inference

By A. Gholami, Sehoon Kim, Zhen Dong, Zi-Mei Yao, Michael W. Mahoney, Kurt Keutzer. 2021

1,633 citations recorded by the source corpus.

As soon as abstract mathematical computations were adapted to computation on digital computers, the problem of efficient representation, manipulation, and communication of the numerical values in those computations arose. Strongly related to the problem of numerical representation is the problem of quantization: in what manner should a set of continuous real-valued numbers be distributed over a fixed discrete set of numbers to minimize the number of bits required and also to maximize the accuracy of the attendant computations? This perennial problem of quantization is particularly relevant whenever memory and/or computational resources are severely restricted, and it has come to the forefront in recent years due to the remarkable performance of Neural Network models in computer vision, natural language processing, and related areas. Moving from floating-point representations to low-precision fixed integer values represented in four bits or less holds the potential to reduce the memory footprint and latency by a factor of 16x; and, in fact, reductions of 4x to 8x are often realized in practice in these applications. Thus, it is not surprising that quantization has emerged recently as an important and very active sub-area of research in the efficient implementation of computations associated with Neural Networks. In this article, we survey approaches to the problem of quantizing the numerical values in deep Neural Network computations, covering the advantages/disadvantages of current methods. With this survey and its organization, we hope to have presented a useful snapshot of the current research in quantization for Neural Networks and to have given an intelligent organization to ease the evaluation of future research in this area.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Evaluation & Benchmarks

Citations in the corpus

Fast Inference of Mixture-of-Experts Language Models with Offloading (2023, background): “A natural way to circumvent this is to reduce the model size through quantization (Nagel et al., 2020; Gholami et al., 2021; Frantar et al., 2022 Generally speaking, the optimal compression rate for most LLMs is 4 bits per parameter Dettmers & Zettlemoyer (2022).”
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022, methodology): “The former quantize models during typically extensive retraining and/or finetuning, using some approximate differentiation mechanism for the rounding operation (Gholami et al., 2021; Nagel et al., 2021).”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “For example, using Hessian information (Dong et al., 2019), step-size quantization (Esser et al., 2019), soft quantization (Gong et al., 2019), mixed-precision via linear programming optimization (Yao et al., 2021), and other learned quantization methods (Zhang et al., 2018; Gholami et al., 2021).”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Please refer to [64] and [23] for more details about the transformer architecture and quantization.”
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (2023, methodology): “Model compression aims to produce more efficient models, using approaches such as pruning, quantization, and knowledge distillation—we refer the reader to the respective surveys [21, 14, 15] for an in-depth discussion.”
AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration (2023, neutral): “…fall into two categories: quantization-aware training (QAT, which relies on backpropagation to update the quantized weights) (Bengio et al., 2013; Gholami et al., 2021; Nagel et al., 2021; Choi et al., 2018) and post-training quantization (Jacob et al., 2018; Nagel et al., 2019; 2020) (PTQ,…”
Large Language Models in Finance: A Survey (2023, background): “In Section 5, we provide a more detailed discussion of which option is more favorable under different circumstances.”
QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models (2023, background): “Quantization is a very popular compression technique, which has seen a vast amount of work (Gholami et al., 2021), especially in the context of LLMs. Specifically, the ability to perform accurate weight quantization for billion-parameter models has greatly boosted their accessibility: it has been…”
Q-Diffusion: Quantizing Diffusion Models (2023, methodology): “Most previous works [12] use the rounding-to-nearest approach, which quantizes deep neural networks by rounding elements w to the nearest quantization values.”
Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation (2021, background): “This postprocessing incurred by nonuniform quantization costs more hardware area and consumes additional energy compared to uniform quantization [1, 13].”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “Quantization refers to the mapping process from floating-point numbers to integers [69].”
How Well Do Sparse ImageNet Models Transfer? (2021, background): “Model compression is a rapidly-developing area, and several generic approaches have been investigated, among which pruning and quantization are among the most popular [16, 25].”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “els via conditional computing [344] and neural network quantization [345] as well as extensive work on improving the specific efficiency of Transformer models, especially”
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (2023, background): “techniques generally fall into two categories: quantization-aware training (QAT, which relies on backpropagation to update the quantized weights) [3, 20, 36, 9] and post-training quantization [24, 35, 34] (PTQ, usually training-free).”
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (2023, neutral): “To recall, rotary position embeddings are a technique that rotates attention head dimensions by an angle that depends on how many tokens are between key and query [SLP + 21].”
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect (2024, methodology): “Model compression methods can be broadly categorized into two categories: quantization Liu et al. (2021); Gholami et al. (2022); Dettmers et al. (2022, 2024) and pruning LeCun et al. (1989); Han et al. (2015).”
F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (2022, methodology): “However, the calculation of INT32 multiplication in these works requires one more operation, which results in extra energy and higher latency (Gholami et al., 2021).”
SqueezeLLM: Dense-and-Sparse Quantization (2023, background): “The first factor is whether retraining is required or not [23].”