ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers

By Zi-Mei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, Yuxiong He. 2022

759 citations recorded by the source corpus.

How to efficiently serve ever-larger trained natural language models in practice has become exceptionally challenging even for powerful cloud servers due to their prohibitive memory/computation requirements. In this work, we present an efficient and affordable post-training quantization approach to compress large Transformer-based models, termed as ZeroQuant. ZeroQuant is an end-to-end quantization and inference pipeline with three main components: (1) a fine-grained hardware-friendly quantization scheme for both weight and activations; (2) a novel affordable layer-by-layer knowledge distillation algorithm (LKD) even without the access to the original training data; (3) a highly-optimized quantization system backend support to remove the quantization/dequantization overhead. As such, we are able to show that: (1) ZeroQuant can reduce the precision for weights and activations to INT8 in a cost-free way for both BERT and GPT3-style models with minimal accuracy impact, which leads to up to 5.19x/4.16x speedup on those models compared to FP16 inference; (2) ZeroQuant plus LKD affordably quantize the weights in the fully-connected module to INT4 along with INT8 weights in the attention module and INT8 activations, resulting in 3x memory footprint reduction compared to the FP16 model; (3) ZeroQuant can be directly applied to two of the largest open-sourced language models, including GPT-J6B and GPT-NeoX20, for which our INT8 model achieves similar accuracy as the FP16 model but achieves up to 5.2x better efficiency.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

QLoRA: Efficient Finetuning of Quantized LLMs (2023, background): “[67] and LLM.int8() [14]) while others use more sophisticated grouping methods [44, 70].”
The case for 4-bit precision: k-bit Inference Scaling Laws (2022, methodology): “Compared to smaller models, LLM quantization poses some unique challenges, such as emergent outliers (Dettmers et al., 2022a; Zeng et al., 2022; Xiao et al., 2022) and optimized low-bit inference for LLMs (Frantar et al., 2022; Park et al., 2022; Yao et al., 2022).”
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (2022, methodology): “A coarse-grained version of per-channel quantization is to use different quantization steps for different channel groups, called group-wise quantization (Shen et al., 2020; Yao et al., 2022).”
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022, methodology): “Figure 1: Quantizing OPT models to 4 and BLOOM models to 3 bit precision, comparing GPTQ with the FP16 baseline and round-to-nearest (RTN) (Yao et al., 2022; Dettmers et al., 2022).”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “ZeroQuant achieves zero-degradation performance for 8-bit quantization of a 20B model.”
AI and Memory Wall (2024, positive): “With current meth-ods, it is relatively easy to quantize inference down to INT4 precision, with minimal impact on accuracy [6, 9, 17, 23, 44].”
QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models (2023, methodology): “Specifically, one approach to do this is by solving a layer-wise quantization problem, stated with respect to W ℓ and the observed calibration data inputs X ℓ at the current layer: Various solvers for Equation (1) have been proposed, with some optimized, in terms of speed and accuracy, particularly ”
Atom: Low-bit Quantization for Efficient and Accurate LLM Serving (2023, methodology): “(Xiao et al., 2023; Shao et al., 2023; Yao et al., 2022; Wei et al., 2023) used mathematical equivalent transformations to manage activation outliers.”
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time (2023, background): “[8] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He.”
Unit Scaling: Out-of-the-Box Low-Precision Training (2023, methodology): “Though recent efforts have been made to improve efficient INT8 quantisation (Yao et al., 2022; Park et al., 2022; Dettmers et al., 2022; Xiao et al., 2022), the use of FP8 enables accelerated inference in the same format as training, promising a substantial improvement in the simplicity and…”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “In the realm of Model Compression, Quantization (Frantar et al., 2023; Dettmers et al., 2022; Kim et al., 2023c; Tao et al., 2022b; Yao et al., 2022; Xiao et al., 2023), Parameter Pruning (Ma et al., 2023d; Zhang et al., 2023h; Frantar and Alistarh, 2023), and Low-Rank Approximation (Xu et al.,…”
LLM-Pruner: On the Structural Pruning of Large Language Models (2023, methodology): “To compress the language model, previous works can be divided into several categories: network pruning [20, 58, 29, 14], knowledge distillation [43, 44, 37], quantization [60, 1, 63] and other techniques, like early exit [57] or dynamic token reduction [61].”
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (2023, methodology): “Similar to [47, 9], we sparsify Transformer layers sequentially in order.”
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (2023, positive): “Furthermore, we delved into the potential of combining the group-wise quantization (Shen et al., 2020a; Sheng et al., 2023a) with LLM-QAT.”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “Compression techniques, such as ZeroQuant [295], are designed, and many existing compression techniques are built-in in the DeepSpeed library for rapid compression, reduced model size, and faster inference.”
High-throughput Generative Inference of Large Language Models with a Single GPU (2023, methodology): “Both spar-sification (Hoefler et al., 2021; Frantar & Alistarh, 2023) and quantization (Kwon et al., 2022; Yao et al., 2022; Park et al., 2022; Xiao et al., 2022; Frantar et al., 2022; Dettmers et al., 2022) have been adopted for LLM inference.”
Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca (2023, methodology): “In the following sections, we will use the 4-bit round-to-nearest (RTN) (Yao et al., 2022; Dettmers et al., 2022) quantized Chinese Alpaca for evaluation, which is more realistic from a user perspective rather than a research-oriented view.”