Data-Free Quantization Through Weight Equalization and Bias Correction

By Markus Nagel, Mart van Baalen, Tijmen Blankevoort, Max Welling. 2019

645 citations recorded by the source corpus.

We introduce a data-free quantization method for deep neural networks that does not require fine-tuning or hyperparameter selection. It achieves near-original model performance on common computer vision architectures and tasks. 8-bit fixed-point quantization is essential for efficient inference on modern deep learning hardware. However, quantizing models to run in 8-bit is a non-trivial task, frequently leading to either significant performance reduction or engineering time spent on training a network to be amenable to quantization. Our approach relies on equalizing the weight ranges in the network by making use of a scale-equivariance property of activation functions. In addition the method corrects biases in the error that are introduced during quantization. This improves quantization accuracy performance, and can be applied to many common computer vision architectures with a straight forward API call. For common architectures, such as the MobileNet family, we achieve state-of-the-art quantized model performance. We further show that the method also extends to other computer vision architectures and tasks such as semantic segmentation and object detection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

The case for 4-bit precision: k-bit Inference Scaling Laws (2022, methodology): “For example, there are methods associated with blocking and grouping (Park et al., 2022; Wu et al., 2020; Jain et al., 2020; Nagel et al., 2019; Krishnamoorthi, 2018; Rusci et al., 2020), centering (Krishnamoorthi, 2018; Jacob et al., 2017), learned data types that are found through clustering…”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Recently, zero-shot quantization [9, 46] and post-training quantization (PTQ) [45, 38] are proposed to address the training-data access and compute requirement challenges since PTQ generally requires no (or minimal) retraining.”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, background): “…(Lin et al., 2016; Jacob et al., 2018; Krishnamoorthi, 2018; Zhang et al., 2018; Choukroun et al., 2019; Dong et al., 2019; Esser et al., 2019; Nagel et al., 2019, 2020), there is relatively little work done on quantizing NLP models (Wang et al., 2018b; Xu et al., 2018), and speciﬁcally on…”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “…63] observe inherent bias in the mean and variance of the weight values following their quantization and propose bias correction methods; and [170, 178] show that equalizing the weight ranges (and implicitly activation ranges) between different layers or channels can reduce quantization errors.”
The Knowledge Within: Methods for Data-Free Model Compression (2019, background): “Previous work [21] showed that 8-bit models can be calibrated without any data if BN layers exist.”
Up or Down? Adaptive Rounding for Post-Training Quantization (2020, methodology): “As Nagel et al. (2019) explained, the practicality of neural network quantization methods is important to take into consideration.”
Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption (2020, neutral): “2018) or quantization (Nagel et al. 2019; Weng et al. 2020).”
A White Paper on Neural Network Quantization (2021, methodology): “Absorbing high biases Nagel et al. (2019) further notice that in some cases, especially after CLE, high biases can lead to differences in the dynamic ranges of the activations.”
SpinQuant: LLM quantization with learned rotations (2024, neutral): “Detailed GPU latency results are provided in the Appendix.”
Loss aware post-training quantization (2019, methodology): “Multiple alternative schemes for the correction of quantization bias were proposed, and those techniques are widely applied in state-of-the-art quantization approaches (Banner et al. 2018; Finkelstein et al. 2019; Nagel et al. 2019; Fang et al. 2020).”
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (2022, positive): “Although our bias and variance correction step applied to YOLO and BERT models is similar to the schemes described in [32] and [1], we now describe our exact procedure for additional clarity: 1.”
A Statistical Framework for Low-bitwidth Training of Deep Neural Networks (2020, background): “Among inference quantization approaches, post training quantization usually does not require access to the partial/full training dataset, and it does not need to re-train/fine-tune the quantized model [1, 2, 3, 4, 5].”
Task-Oriented Feature Distillation (2020, methodology): “Various techniques have been proposed to tackle this issue, including pruning [15, 41, 72, 67, 39], quantization [35, 46, 11, 50], lightweight model de-sign [23, 49, 27], and knowledge distillation (KD) [22, 53, 70].”
Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation (2020, background): “This is commonly referred to as “per-channel” weight quantization in literature and we follow that convention [21, 25, 26, 38, 46].”
Any-Precision Deep Neural Networks (2019, background): “[22] identiﬁed two issues leading to the large accuracy drop, the large variation in the weight ranges across channels and biased output errors due to quantization errors affecting following layers.”
BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction (2021, background): “(16)); if Activation Quantization is triggered then Update the activation quantization step size (Eq.”
Accurate Post Training Quantization With Small Calibration Sets (2021, background): “Therefore, much attention has recently been dedicated to post-training quantization methods (Nagel et al., 2019; Banner et al., 2018; Zhao et al., 2019), which can be more easily applied in practice.”
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (2023, background): “techniques generally fall into two categories: quantization-aware training (QAT, which relies on backpropagation to update the quantized weights) [3, 20, 36, 9] and post-training quantization [24, 35, 34] (PTQ, usually training-free).”
CrypTFlow2: Practical 2-Party Secure Inference (2020, methodology): “The ML models used by all prior works on secure inference are expressed using fixed-point arithmetic; such models can be obtained from [39, 42, 45, 52].”