The case for 4-bit precision: k-bit Inference Scaling Laws

By Tim Dettmers, Luke S. Zettlemoyer. 2022

338 citations recorded by the source corpus.

Quantization methods reduce the number of bits required to represent each parameter in a model, trading accuracy for smaller memory footprints and inference latencies. However, the final model size depends on both the number of parameters of the original model and the rate of compression. For example, a 30B 8-bit model and a 60B 4-bit model have the same number of bits but may have very different zero-shot accuracies. In this work, we study this trade-off by developing inference scaling laws of zero-shot performance in Large Language Models (LLMs) to determine the bit-precision and model size that maximizes zero-shot performance. We run more than 35,000 experiments with 16-bit inputs and k-bit parameters to examine which zero-shot quantization methods improve scaling for 3 to 8-bit precision at scales of 19M to 176B parameters across the LLM families BLOOM, OPT, NeoX/Pythia, and GPT-2. We find that it is challenging to improve the bit-level scaling trade-off, with the only improvements being the use of a small block size -- splitting the parameters into small independently quantized blocks -- and the quantization data type being used (e.g., Int vs Float). Overall, our findings show that {4-bit} precision is almost universally optimal for total model bits and zero-shot accuracy.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Efficiency & Compression

Citations in the corpus

Fast Inference of Mixture-of-Experts Language Models with Offloading (2023, methodology): “To use these LLMs on more affordable hardware setups, one must either compress model parameters (Dettmers et al., 2022; Frantar et al., 2022) or offload parameters to a cheaper storage, be it RAM or SSD (Pudipeddi et al., 2020; Sheng et al., 2023).”
QLoRA: Efficient Finetuning of Quantized LLMs (2023, methodology): “In line with previous work on quantization [13], our MMLU and Elo results indicate that with a given finetuning and inference resource budget it is beneficial to increase the number of parameters in the base model while decreasing their precision.”
Stable and low-precision training for large-scale vision-language models (2023, background): “Moreover, while int8 and int4 are used for inference [16, 64, 15], and there is earlier work exploring 8 bit training for convnets [59, 77, 10], these formats are not commonly used for training transformer models at scale.”
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (2023, positive): “Current techniques for LLM compression can shrink these models further by a factor of about 4x, while preserving their performance [DLBZ22, XLS + 22, FAHA22, DZ22].”
QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models (2023, methodology): “On large models (Dettmers et al., 2022; Dettmers & Zettle-moyer, 2022), in particular also MoEs (Kim et al., 2022b; Yi et al., 2023), just simple rounding can decrease precision to 8 or even 4 bits per weight, at minimal accuracy loss relative to the standard half (16-bit) precision employed for…”
High-throughput Generative Inference of Large Language Models with a Single GPU (2023, background): “A concurrent work (Dettmers & Zettlemoyer, 2022) also finds that 4-bit precision is almost optimal for total model bits and zero-shot accuracy on OPT models.”
AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration (2023, positive): “It is an effective method to reduce the model size and inference costs of LLMs (Dettmers et al., 2022; Frantar et al., 2022; Yao et al., 2022; Xiao et al., 2022).”
A Survey on Model Compression for Large Language Models (2023, neutral): “FLOPs [Dettmers and Zettlemoyer, 2022; Yuan et al., 2023; Wei et al., 2023] measures the number of arithmetic operations involving floating-point numbers (typically 32-bit or 16-bit) that the LLM performs when processing input data.”
QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks (2024, background): “This directly refutes Dettmers & Zettlemoyer (2023)’s claim that 4-bit models are “optimal” and indicates that as the field of PTQ develops, 2-bit models are likely to scale better than 3-bit models in the near future.”
OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models (2023, methodology): “Consider the quantized object, exiting LLM quantization can be classified into two fields: weightonly quantization (Frantar et al. (2022); Lin et al.”
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (2023, methodology): “Very recent work by Dettmers and Zettlemoyer [4] investigate scaling relationships between model size, quantization bits, and different notions of accuracy for massive LLMs, observing a high degree of correlation between perplexity scores and aggregated zero-shot accuracy across tasks, as well as sa”