Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models

By Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fisher Yu, Xianglong Liu. 2022

229 citations recorded by the source corpus.

Transformer architecture has become the fundamental element of the widespread natural language processing~(NLP) models. With the trends of large NLP models, the increasing memory and computation costs hinder their efficient deployment on resource-limited devices. Therefore, transformer quantization attracts wide research interest. Recent work recognizes that structured outliers are the critical bottleneck for quantization performance. However, their proposed methods increase the computation overhead and still leave the outliers there. To fundamentally address this problem, this paper delves into the inherent inducement and importance of the outliers. We discover that $\boldsymbol \gamma$ in LayerNorm (LN) acts as a sinful amplifier for the outliers, and the importance of outliers varies greatly where some outliers provided by a few tokens cover a large area but can be clipped sharply without negative impacts. Motivated by these findings, we propose an outlier suppression framework including two components: Gamma Migration and Token-Wise Clipping. The Gamma Migration migrates the outlier amplifier to subsequent modules in an equivalent transformation, contributing to a more quantization-friendly model without any extra burden. The Token-Wise Clipping takes advantage of the large variance of token range and designs a token-wise coarse-to-fine pipeline, obtaining a clipping range with minimal final quantization loss in an efficient way. This framework effectively suppresses the outliers and can be used in a plug-and-play mode. Extensive experiments prove that our framework surpasses the existing works and, for the first time, pushes the 6-bit post-training BERT quantization to the full-precision (FP) level. Our code is available at https://github.com/wimh966/outlier_suppression.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Efficiency & Compression

Citations in the corpus

The case for 4-bit precision: k-bit Inference Scaling Laws (2022, methodology): “Large Language Models (LLMs) are widely adopted for zero/few-shot inference (Zhang et al., 2022; Black et al., 2022; Zeng et al., 2022; Scao et al., 2022), but they can be challenging to use both due to their large memory footprints – up to 352 GB of GPU memory for 175B models – and high latency.”
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (2022, methodology): “Experiments show that SmoothQuant is hardware-efficient: it can maintain the performance of OPT-175B (Zhang et al., 2022), BLOOM-176B (Scao et al., 2022) , GLM-130B (Zeng et al., 2022), and MT-NLG 530B (Smith et al., 2022), leading to up to 1.51 × speed up and 1.96 × memory saving on PyTorch.”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “Large magnitude outlier features in language models have been studied before (Timkey and van Schijndel, 2021; Bondarenko et al., 2021; Wei et al., 2022; Luo et al., 2021).”
A Simple and Effective Pruning Approach for Large Language Models (2023, background): “Our work is also related to recent studies on the existence of large magnitude outlier features in large language models (Kovaleva et al., 2021; Bondarenko et al., 2021; Timkey & Schijndel, 2021; Luo et al., 2021; Puccetti et al., 2022; Wei et al., 2022b).”
SqueezeLLM: Dense-and-Sparse Quantization (2023, unknown): “Notably, [12] proposes to retain outlier activations as floating-point representations, while [69] suggests migrating outlier factors to subsequent layers without altering the functionality.”
AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration (2023, unknown): “…cached from a small calibration set (we take a small calibration set from he pre-training dataset in order not to overfit to a specific task). s is a per-(input) channel scaling factor; for s − 1 · X , it can usually be fused into the previous operator (Wei et al., 2022b; Xiao et al., 2022).”
QuIP: 2-Bit Quantization of Large Language Models With Guarantees (2023, background): “[28] Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu.”
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (2024, unknown): “Several works show that, unlike weights, quantizing the activations is hard due to the outlier features [Wei et al., 2022, Dettmers et al., 2022, Xiao et al., 2023].”
SpinQuant: LLM quantization with learned rotations (2024, unknown): “Detailed GPU latency results are provided in the Appendix.”
OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models (2023, methodology): “Considering that outliers in the activation map are systematic and unique to specific channels, previous methods such as SmoothQuant (Xiao et al. (2023)) migrate the difficulty of quantization from activations to weights with a mathematically equivalent transformation. However, they hand-craft the e”