A White Paper on Neural Network Quantization

By Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, Tijmen Blankevoort. 2021

894 citations recorded by the source corpus.

While neural networks have advanced the frontiers in many applications, they often come at a high computational cost. Reducing the power and latency of neural network inference is key if we want to integrate modern networks into edge devices with strict power and compute requirements. Neural network quantization is one of the most effective ways of achieving these savings but the additional noise it induces can lead to accuracy degradation. In this white paper, we introduce state-of-the-art algorithms for mitigating the impact of quantization noise on the network's performance while maintaining low-bit weights and activations. We start with a hardware motivated introduction to quantization and then consider two main classes of algorithms: Post-Training Quantization (PTQ) and Quantization-Aware-Training (QAT). PTQ requires no re-training or labelled data and is thus a lightweight push-button approach to quantization. In most cases, PTQ is sufficient for achieving 8-bit quantization with close to floating-point accuracy. QAT requires fine-tuning and access to labeled training data but enables lower bit quantization with competitive results. For both solutions, we provide tested pipelines based on existing literature and extensive experimentation that lead to state-of-the-art performance for common deep learning models and tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022, neutral): “The former quantize models during typically extensive retraining and/or ﬁne-tuning, using some approximate differentiation mechanism for the rounding operation (Gholami et al., 2021; Nagel et al., 2021).”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, background): “For a more comprehensive overview of neural network quantization, please refer to Nagel et al. (2021).”
Unit Scaling: Out-of-the-Box Low-Precision Training (2023, neutral): “This process typically results in degraded accuracy, requiring additional techniques such as quantisation-aware training (see Nagel et al. (2021) for a thorough discussion on this topic).”
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (2023, background): “techniques generally fall into two categories: quantization-aware training (QAT, which relies on backpropagation to update the quantized weights) [3, 20, 36, 9] and post-training quantization [24, 35, 34] (PTQ, usually training-free).”
Overcoming Oscillations in Quantization-Aware Training (2022, methodology): “The STE gradient approximation has been widely adopted in recent literature closing the gap between quantized and fullprecision accuracy for a wide range of tasks and networks (Gupta et al., 2015; Zhou et al., 2016; Krishnamoorthi, 2018; Jain et al., 2019; Esser et al., 2020; Bhalgat et al., 2020; N”
Q-Diffusion: Quantizing Diffusion Models (2023, methodology): “This work explores the quantization [50, 8, 49, 7, 29] of the noise estimation model used in the diffusion model to accelerate the denoising of all time steps.”
QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models (2023, methodology): “Kim et al. (2022a) achieve 2-bit quantization on a 5 billion parameter MoE, which is considered relatively small in this area, by further optimization of the model via Quantization-Aware Training (Nagel et al., 2021).”
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (2022, neutral): “AdaQuant [19] relaxes the AdaRound constraint, allowing weights to change during quantization-aware optimization, via straight-through estimation [33].”
Seed-TTS: A Family of High-Quality Versatile Speech Generation Models (2024, methodology): “…applied methods to reduce the memory and computation consumption on the language model side, such as grouped-query attention [Ainslie et al., 2023], paged attention [Kwon et al., 2023], flash attention [Dao et al., 2022, Dao, 2023], and model quantization [Nagel et al., 2021, Guo et al., 2024].”
Post-Training Quantization on Diffusion Models (2022, neutral): “Therefore, we propose to introduce post-training quantization (PTQ) [21, 3, 16] into DM acceleration.”
AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration (2023, neutral): “…quantization-aware training (QAT, which relies on backpropagation to update the quantized weights) (Bengio et al., 2013; Gholami et al., 2021; Nagel et al., 2021; Choi et al., 2018) and post-training quantization (Jacob et al., 2018; Nagel et al., 2019; 2020) (PTQ, usually training-free).”
F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (2022, methodology): “…methods, such as network pruning (Tanaka et al., 2020; Li et al., 2021; Ma et al., 2020; 2021a; Yuan et al., 2021) and slimming (Liu et al., 2017; 2018), as it achieves a large compression ratio (Krishnamoorthi, 2018; Nagel et al., 2021) and is computationally beneﬁcial for integer-only hardware.”