Understanding and Overcoming the Challenges of Efficient Transformer Quantization

By Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort. 2021

196 citations recorded by the source corpus.

Transformer-based architectures have become the de-facto standard models for a wide range of Natural Language Processing tasks. However, their memory footprint and high latency are prohibitive for efficient deployment and inference on resource-limited devices. In this work, we explore quantization for transformers. We show that transformers have unique quantization challenges – namely, high dynamic activation ranges that are difficult to represent with a low bit fixed-point format. We establish that these activations contain structured outliers in the residual connections that encourage specific attention patterns, such as attending to the special separator token. To combat these challenges, we present three solutions based on post-training quantization and quantization-aware training, each with a different set of compromises for accuracy, model size, and ease of use. In particular, we introduce a novel quantization scheme – per-embedding-group quantization. We demonstrate the effectiveness of our methods on the GLUE benchmark using BERT, establishing state-of-the-art results for post-training quantization. Finally, we show that transformer weights and embeddings can be quantized to ultra-low bit-widths, leading to significant memory savings with a minimum accuracy loss. Our source code is available at https://github.com/qualcomm-ai-research/transformer-quantization.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

The case for 4-bit precision: k-bit Inference Scaling Laws (2022, background): “…Quantization Through Proxy Quantization Outlier features that emerge in large language models (Gao et al., 2019; Timkey & van Schijndel, 2021; Bondarenko et al., 2021; Wei et al., 2022; Luo et al., 2021; Kovaleva et al., 2021; Puccetti et al., 2022) can cause large quantization errors and…”
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (2022, methodology): “Due to the persistence of outliers and the small variance inside each channel, if we could perform per-channel quantization (Bondarenko et al., 2021) of the activation (i.”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “Large magnitude outlier features in language models have been studied before (Timkey and van Schijndel, 2021; Bondarenko et al., 2021; Wei et al., 2022; Luo et al., 2021).”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, methodology): “In this paper, we present ZeroQuant, an end-to-end post-training quantization and inference pipeline, to address those challenges, targeting both INT8 and INT4/INT8 mixed-precision quantization.”
The Efficiency Misnomer (2021, background): “Memory consumption is one of the main dimensions of efficiency and it has been used as a cost indicator in various research works (Bondarenko et al., 2021; Dosovitskiy et al., 2020; Kondratyuk et al., 2021).”
Unit Scaling: Out-of-the-Box Low-Precision Training (2023, background): “These are large-magnitude values concentrated in particular sequenceelements (Bondarenko et al., 2021) or feature-dimensions (Dettmers et al.”
A Simple and Effective Pruning Approach for Large Language Models (2023, result): “Our work is also related to recent studies on the existence of large magnitude outlier features in large language models (Kovaleva et al., 2021; Bondarenko et al., 2021; Timkey & Schijndel, 2021; Luo et al., 2021; Puccetti et al., 2022; Wei et al., 2022b).”
Outliers Dimensions that Disrupt Transformers Are Driven by Frequency (2022, background): “Outlier dimensions have also been found to make model quantization challenging (Bondarenko et al., 2021; Dettmers et al., 2022) as they need to be treated separately from others when defining quantization schemes.”
Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models (2022, methodology): “For PTQ, we compare with the prevalent calibration mechanisms including MinMax [39], OMSE [28], Percentile [29], EasyQuant [40] and PEG [26].”
OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networks (2022, background): “However, these all come at the cost of decreasing the accuracy of the model, or increasing the time it takes to train it, or both [49, 7, 57].”
SqueezeLLM: Dense-and-Sparse Quantization (2023, methodology): “Another challenge when quantizing weights in LLMs into low-bit precision involves outlier values [3, 12, 68, 69].”
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (2024, background): “Other works reconsidered the di-mension along which we quantize in order to reduce quantization error (or else added per-channel compensation to improve quantization performance) [2, 14, 32, 33].”