SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

By Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, Song Han. 2022

1,734 citations recorded by the source corpus.

Large language models (LLMs) show excellent performance but are compute- and memory-intensive. Quantization can reduce memory and accelerate inference. However, existing methods cannot maintain accuracy and hardware efficiency at the same time. We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit activation (W8A8) quantization for LLMs. Based on the fact that weights are easy to quantize while activations are not, SmoothQuant smooths the activation outliers by offline migrating the quantization difficulty from activations to weights with a mathematically equivalent transformation. SmoothQuant enables an INT8 quantization of both weights and activations for all the matrix multiplications in LLMs, including OPT, BLOOM, GLM, MT-NLG, Llama-1/2, Falcon, Mistral, and Mixtral models. We demonstrate up to 1.56x speedup and 2x memory reduction for LLMs with negligible loss in accuracy. SmoothQuant enables serving 530B LLM within a single node. Our work offers a turn-key solution that reduces hardware costs and democratizes LLMs. Code is available at https://github.com/mit-han-lab/smoothquant.

Subfields: Artificial Intelligence, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Efficiency & Compression

Citations in the corpus

QLoRA: Efficient Finetuning of Quantized LLMs (2023, background): “[67] and LLM.int8() [14]) while others use more sophisticated grouping methods [44, 70].”
Stable and low-precision training for large-scale vision-language models (2023, background): “Moreover, while int8 and int4 are used for inference [17, 66, 16], and there is earlier work exploring 8 bit training for convnets [61, 79, 10], these formats are not commonly used for training transformer models at scale.”
The case for 4-bit precision: k-bit Inference Scaling Laws (2022, methodology): “The choice of these particular zero-shot tasks was mainly motivated by previous work (Dettmers et al., 2022a; Yao et al., 2022; Xiao et al., 2022).”
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (2024, methodology): “SmoothQuant (Xiao et al., 2023a) is a post-training quantization method that is more closely related to our work.”
A Simple and Effective Pruning Approach for Large Language Models (2023, methodology): “The fast pace of LLM quantization research (Dettmers et al., 2022; Frantar et al., 2023a; Xiao et al., 2023; Ahmadian et al., 2023) has led to substantial resource savings for these models (Sheng et al.”
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time (2023, background): “[14] Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han.”
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time (2023, methodology): “Recently, there is active research attempting to apply one or a combination of such techniques in LLM inference (Yao et al., 2022; Park et al., 2022; Dettmers et al., 2022; Frantar et al., 2022; Frantar & Alistarh, 2023; Bansal et al., 2022; Xiao et al., 2022).”
AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration (2023, positive): “It is an effective method to reduce the model size and inference costs of LLMs (Dettmers et al., 2022; Frantar et al., 2022; Yao et al., 2022; Xiao et al., 2022).”
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (2023, neutral): “Quantization function We illustrate our quantized transformer model in Figure 3.”
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (2023, background): “For example, quantization reduces the numeric precision of model weights and activations and speeds up training and inference (Dettmers et al., 2022; 2023; Xiao et al., 2023).”
A Comprehensive Overview of Large Language Models (2023, positive): “Importantly, MLLMs align closely with human perceptual experiences, leveraging the synergistic nature of our multisensory inputs to form a comprehensive understanding of the world [207], [188].”
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (2023, methodology): “W8A8 quantization [12, 54], which maps both activation and weights into INT8; 2.”
FastComposer: Tuning-Free Multi-subject Image Generation with Localized Attention (2023, neutral): “Our design allows the generation of images featuring specified subjects with only forward passes and can be further integrated with model compression techniques [2, 11, 40] to boost deployment efficiency.”
OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models (2023, methodology): “…can be attributed to the fact that these methods (Frantar et al., 2022; Lin et al., 2023; Wei et al., 2023) primarily rely on handcrafted quantization parameters such as migration strength (Xiao et al., 2023) and scaling parameters (Wei et al., 2023), which often leads to lower performance.”
SpinQuant: LLM quantization with learned rotations (2024, methodology): “There exist many ways to mitigate the effect of outliers [46, 11].”
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (2023, methodology): “This is particularly problematic for massive language models, as they are known to have a small number of highly sensitive outlier features [3, 46].”