GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

By Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh. 2022

2,286 citations recorded by the source corpus.

Generative Pre-trained Transformer models, known as GPT or OPT, set themselves apart through breakthrough performance across complex language modelling tasks, but also by their extremely high computational and storage costs. Specifically, due to their massive size, even inference for large, highly-accurate GPT models may require multiple performant GPUs, which limits the usability of such models. While there is emerging work on relieving this pressure via model compression, the applicability and performance of existing compression techniques is limited by the scale and complexity of GPT models. In this paper, we address this challenge, and propose GPTQ, a new one-shot weight quantization method based on approximate second-order information, that is both highly-accurate and highly-efficient. Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline. Our method more than doubles the compression gains relative to previously-proposed one-shot quantization methods, preserving accuracy, allowing us for the first time to execute an 175 billion-parameter model inside a single GPU for generative inference. Moreover, we also show that our method can still provide reasonable accuracy in the extreme quantization regime, in which weights are quantized to 2-bit or even ternary quantization levels. We show experimentally that these improvements can be leveraged for end-to-end inference speedups over FP16, of around 3.25x when using high-end GPUs (NVIDIA A100) and 4.5x when using more cost-effective ones (NVIDIA A6000). The implementation is available at https://github.com/IST-DASLab/gptq.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Efficiency & Compression

Citations in the corpus

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023, background): “Quantization (Dettmers et al., 2022; Zeng et al., 2022; Frantar et al., 2022) reduces the size of weights and activations, including keys and values, by lowering precision.”
Fast Inference of Mixture-of-Experts Language Models with Offloading (2023, methodology): “To use these LLMs on more affordable hardware setups, one must either compress model parameters (Dettmers et al., 2022; Frantar et al., 2022) or offload parameters to a cheaper storage, be it RAM or SSD (Pudipeddi et al., 2020; Sheng et al., 2023).”
QLoRA: Efficient Finetuning of Quantized LLMs (2023, background): “Lossy quantization approaches study the trade-offs for regular rounding [13, 72, 48] or how to optimize rounding decisions to improve quantization precision [18].”
Stable and low-precision training for large-scale vision-language models (2023, background): “Another line of work studies large models inference with more than 250M parameters by considering 16-bit inputs and k-bit weights [16, 24, 68].”
The case for 4-bit precision: k-bit Inference Scaling Laws (2022, methodology): “While we do not study learned data types in depth, we are the first work that shows that these are critical for improving bit-level scaling for LLMs. Scaling Laws for Inference.”
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (2022, background): “Quantization of LLMs. GPTQ (Frantar et al., 2022) applies quantization only to weights but not activations (please find a short discussion in Appendix A).”
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024, methodology): “We recommend future work explores compression techniques (Treviso et al., 2023) such as quantization (Dettmers et al., 2022; Frantar et al., 2022; Ahmadian et al., 2023) and pruning (Frantar and Alistarh, 2023; Ogueji et al., 2022; Gale et al., 2019; Ahia et al., 2021) for multilingual…”
NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024, positive): “For weight quantization stage, we adopt GPTQ (Frantar et al., 2022), a post-training weight quantization method that utilizes approximate Hessian information to reduce the precision of the weights.”
A Survey on Model Compression for Large Language Models (2023, positive): “GPTQ [ Frantar et al. , 2022 ] acknowl-edges that the methods mentioned above work well for low compression targets like 8-bit weights, but face challenges in maintaining accuracy at higher rates.”
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time (2023, background): “[11] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.”
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (2024, background): “Several recent works have also shown that the model sizes can be compressed significantly using quantization [40, 41,43,73].”
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time (2023, methodology): “…the influence of activation outliers; SmoothQuant (Xiao et al., 2022) enables efficient 8-bit weight and activation for LLM inference; GPTQ (Frantar et al., 2022) adopts a one-shot weight quantization method based on approximate second-order information for accuracy and efficiency;…”
Atom: Low-bit Quantization for Efficient and Accurate LLM Serving (2023, methodology): “We also incorporate GPTQ (Frantar et al., 2023) when quantizing the weight matrix since this is purely an offline process and offers an accuracy boost without sacrificing throughput.”
Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models (2025, neutral): “Unlike model compression techniques such as quantization [29,50,97] or KV cache compression [55, 76, 107, 110], which focus on reducing model size for lightweight inference, efficient reasoning in LLMs emphasizes smart and concise reasoning by optimizing the length of generated reasoning sequences…”
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (2023, positive): “Additionally, alternative approaches explore quantization methods specifically tailored to LLMs, as discussed in (Frantar et al., 2022; Xiao et al., 2022; Yao et al., 2022; Dettmers et al., 2022b;a).”
A Comprehensive Overview of Large Language Models (2023, neutral): “Importantly, MLLMs align closely with human perceptual experiences, leveraging the synergistic nature of our multisensory inputs to form a comprehensive understanding of the world [207], [188].”
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (2023, methodology): “The decoding stage dominates LLMs’ total runtime [17, 32], which is highly memory-bounded with a single batch size.”