Up or Down? Adaptive Rounding for Post-Training Quantization

By Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, Tijmen Blankevoort. 2020

883 citations recorded by the source corpus.

When quantizing neural networks, assigning each floating-point weight to its nearest fixed-point value is the predominant approach. We find that, perhaps surprisingly, this is not the best we can do. In this paper, we propose AdaRound, a better weight-rounding mechanism for post-training quantization that adapts to the data and the task loss. AdaRound is fast, does not require fine-tuning of the network, and only uses a small amount of unlabelled data. We start by theoretically analyzing the rounding problem for a pre-trained neural network. By approximating the task loss with a Taylor series expansion, the rounding task is posed as a quadratic unconstrained binary optimization problem. We simplify this to a layer-wise local loss and propose to optimize this loss with a soft relaxation. AdaRound not only outperforms rounding-to-nearest by a significant margin but also establishes a new state-of-the-art for post-training quantization on several networks and tasks. Without fine-tuning, we can quantize the weights of Resnet18 and Resnet50 to 4 bits while staying within an accuracy loss of 1%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Fast Inference of Mixture-of-Experts Language Models with Offloading (2023, background): “A natural way to circumvent this is to reduce the model size through quantization (Nagel et al., 2020; Gholami et al., 2021; Frantar et al., 2022 Generally speaking, the optimal compression rate for most LLMs is 4 bits per parameter Dettmers & Zettlemoyer (2022).”
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022, neutral): “The AdaRound method (Nagel et al., 2020) computes a data-dependent rounding by annealing a penalty term, which encourages weights to move towards grid points corresponding to quantization levels.”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Recently, zero-shot quantization [9, 46] and post-training quantization (PTQ) [45, 38] are proposed to address the training-data access and compute requirement challenges since PTQ generally requires no (or minimal) retraining.”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, methodology): “…(Lin et al., 2016; Jacob et al., 2018; Krishnamoorthi, 2018; Zhang et al., 2018; Choukroun et al., 2019; Dong et al., 2019; Esser et al., 2019; Nagel et al., 2019, 2020), there is relatively little work done on quantizing NLP models (Wang et al., 2018b; Xu et al., 2018), and speciﬁcally on…”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Another notable work is AdaRound [177] which shows that the naive round-to-nearest method for quantization can counter-intuitively results in sub-optimal solutions, and it proposes an adaptive rounding method that better reduces the loss.”
BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction (2021, methodology): “Here we adopt adaptive rounding (Nagel et al., 2020) for weights and learned step size (Esser et al.”
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (2022, neutral): “Further, most works [42, 31, 19] focus on compressing linear and convolutional layers, which can be unfolded into linear ones, as these are prevalent in practice, and use the squared loss to measure the approximation error.”
Post-Training Quantization for Vision Transformer (2021, background): “[20] propose AdaRound, a weight-rounding mechanism for post-training quantization that adapts to the data and the task loss.”
Bayesian Bits: Unifying Quantization and Pruning (2020, methodology): “(b) Bayesian Bits results on MobileNet V2, compared to AdaRound [27], LSQ [8], and TQT [15] * results run by the authors.”
Implicit Diffusion Models for Continuous Super-Resolution (2023, neutral): “Extensive experiments illustrate that our IDM exhibits state-of-the-art performance.”
Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming (2020, methodology): “In contrast to AdaRound (Nagel et al., 2020) which restricted the change of the weights to be within ± 1 we allow the weights to change as needed.”
A White Paper on Neural Network Quantization (2021, methodology): “However, Nagel et al. (2020) showed that rounding-to-nearest is not optimal in terms of the task loss when quantizing weights in the post-training regime.”
nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models (2022, background): “Note that various uniform quantization methods are being intensively studied because uniform quantization can be implemented by simple ﬁxed-point operations [24], [27], [28], [33].”
Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models (2022, background): “Among them, quantization [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], working in the low-precision arithmetic fashion, is one of the key approaches to compress large models and fit them into the lightweight devices.”
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (2023, methodology): “techniques generally fall into two categories: quantization-aware training (QAT, which relies on backpropagation to update the quantized weights) [3, 20, 36, 9] and post-training quantization [24, 35, 34] (PTQ, usually training-free).”
Extreme Compression for Pre-trained Transformers Made Simple and Efficient (2022, methodology): “Popular quantization methods include post-training quantization [45, 33, 29], quantization-aware training (QAT) [23, 4, 31, 22], and their variations [44, 24, 12].”
QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks (2024, methodology): “In QuIP # , we follow existing state-of-the-art PTQ methods and round weights to minimize the per-layer proxy loss, as QuIP # formalized by Nagel et al. (2020): Here, W ∈ R m × n is the original weight matrix in a linear layer, ˆ W ∈ R m × n are the quantized weights, x ∈ R n is an input vector…”
QuIP: 2-Bit Quantization of Large Language Models With Guarantees (2023, methodology): “Nagel et al. [20] are the first to motivate the “adaptive rounding” proxy objective (Eq.”