LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

By Tim Dettmers, Michael Lewis, Younes Belkada, Luke S. Zettlemoyer. 2022

1,133 citations recorded by the source corpus.

Large language models have been widely adopted but require significant GPU memory for inference. We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for inference by half while retaining full precision performance. With our method, a 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation. This is made possible by understanding and working around properties of highly systematic emergent features in transformer language models that dominate attention and transformer predictive performance. To cope with these features, we develop a two-part quantization procedure, LLM.int8(). We first use vector-wise quantization with separate normalization constants for each inner product in the matrix multiplication, to quantize most of the features. However, for the emergent outliers, we also include a new mixed-precision decomposition scheme, which isolates the outlier feature dimensions into a 16-bit matrix multiplication while still more than 99.9% of values are multiplied in 8-bit. Using LLM.int8(), we show empirically it is possible to perform inference in LLMs with up to 175B parameters without any performance degradation. This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our software.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Hardware and Architecture, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Hardware and Architecture

In Large Language Models

In Efficiency & Compression

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “Recently, Dettmers et al. (2022) identified a phase transition for models larger than 6.7B, in which the emergence of “outliers features” is observed.”
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023, background): “Quantization (Dettmers et al., 2022; Zeng et al., 2022; Frantar et al., 2022) reduces the size of weights and activations, including keys and values, by lowering precision.”
Efficiently Scaling Transformer Inference (2022, neutral): “…(Sanh et al., 2019; Sun et al., 2020), and model compression techniques, such as pruning (Li et al., 2020b; Brix et al., 2020; Zhou et al., 2021; Li et al., 2020a; Wang et al., 2020), or quantization (Dettmers et al., 2022; Abdolrashidi et al., 2021; Zafrir et al., 2019; Zhang et al., 2018).”
Stable and low-precision training for large-scale vision-language models (2023, methodology): “This suggest why zero-init layer scale enables float8 training—zero-init layer scale prevents high feature magnitudes which may cause issues for low precision training [16].”
The case for 4-bit precision: k-bit Inference Scaling Laws (2022, methodology): “The choice of these particular zero-shot tasks was mainly motivated by previous work (Dettmers et al., 2022a; Yao et al., 2022; Xiao et al., 2022).”
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (2022, methodology): “int8() (Dettmers et al., 2022), and Outlier Suppression (Wei et al.”
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022, positive): “int8() [5], and nuQmm [24]—employ relatively simple quantization schemes based on rounding to the nearest (RTN) quantization level.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “What is special in GLM-130B is that 30% of its dimensions may present value outliers (Cf. Figure 12), while other GPT-based LLMs (e.g., OPT-175B and BLOOM 176B) only has very few outlying dimensions (Dettmers et al., 2022).”
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023, background): “The inference time per token is within 13ms (1.61 seconds / 128 tokens).”
Accelerating Large Language Model Decoding with Speculative Sampling (2023, methodology): “Since sampling performance is heavily coupled with the model size in memory, quantisation to int8 or even int4 (Dettmers et al., 2022; Yao et al., 2022) and distillation (Jiao et al., 2020; Sanh et al., 2019) of transformers are eﬀective techniques for reducing sampling latency with little to no…”
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (2024, result): “The above observation for key cache aligns with previous findings that certain fixed columns in activations exhibit larger outliers (Dettmers et al., 2022; Lin et al., 2023).”
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (2023, methodology): “This is particularly problematic for massive language models, as they are known to have a small number of highly sensitive outlier features [3, 46].”
SliceGPT: Compress Large Language Models by Deleting Rows and Columns (2024, methodology): “Following similar work (Frantar & Alistarh, 2023; Dettmers et al., 2022; Frantar et al., 2022; Dettmers et al., 2023), we use the LM Evaluation Harness (Gao et al., 2021) with default parameters in our evaluations.”
MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data (2023, positive): “When combined with LoRA, this strategy further reduces the memory needed for training [13] [14].”
Sparse Autoencoders Find Highly Interpretable Features in Language Models (2023, background): “…after a non-linearity in an MLP), it has been noted that there is a tendency for transformers to store information in the residual stream basis(Dettmers et al., 2022), which is believed to be caused by the Adam optimiser saving gradients with finite precision in the residual basis(Elhage et…”