SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression

By Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, Dan Alistarh. 2023

444 citations recorded by the source corpus.

Recent advances in large language model (LLM) pretraining have led to high-quality LLMs with impressive abilities. By compressing such LLMs via quantization to 3-4 bits per parameter, they can fit into memory-limited devices such as laptops and mobile phones, enabling personalized use. However, quantization down to 3-4 bits per parameter usually leads to moderate-to-high accuracy losses, especially for smaller models in the 1-10B parameter range, which are well-suited for edge deployments. To address this accuracy issue, we introduce the Sparse-Quantized Representation (SpQR), a new compressed format and quantization technique which enables for the first time near-lossless compression of LLMs across model scales, while reaching similar compression levels to previous methods. SpQR works by identifying and isolating outlier weights, which cause particularly-large quantization errors, and storing them in higher precision, while compressing all other weights to 3-4 bits, and achieves relative accuracy losses of less than 1% in perplexity for highly-accurate LLaMA and Falcon LLMs. This makes it possible to run 33B parameter LLM on a single 24 GB consumer GPU without any performance degradation at 15% speedup thus making powerful LLMs available to consumer without any downsides. SpQR comes with efficient algorithms for both encoding weights into its format, as well as decoding them efficiently at runtime. Specifically, we provide an efficient GPU inference algorithm for SpQR which yields faster inference than 16-bit baselines at similar accuracy, while enabling memory compression gains of more than 4x.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Efficiency & Compression

Citations in the corpus

Fast Inference of Mixture-of-Experts Language Models with Offloading (2023, critical): “While there are more extreme algorithms for 3-and even 2-bit compression Chee et al. (2023); Lin et al. (2023); Dettmers et al. (2023), they are typically inferior to choosing a smaller model and quantizing it to around 4 bits.”
SliceGPT: Compress Large Language Models by Deleting Rows and Columns (2024, methodology): “Following similar work (Frantar & Alistarh, 2023; Dettmers et al., 2022; Frantar et al., 2022; Dettmers et al., 2023), we use the LM Evaluation Harness (Gao et al., 2021) with default parameters in our evaluations.”
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (2024, unknown): “This demonstrates the benefits of our approach for enabling long sequence length inference.”
QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models (2023, background): “Pushing towards even lower bitwidths via more sophisticated compression formats, like multi-level grouping coupled with higher-precision outliers (Dettmers et al., 2023b), or new quantization techniques, like incoherence preprocessing (Chee et al., 2023), is an active area of research.”
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect (2024, methodology): “MiniLLM Gu et al. (2023) uses reverse Kullback-Leibler divergence to prevent the student model from overestimating the low-probability regions of the teacher distribution.”
SqueezeLLM: Dense-and-Sparse Quantization (2023, methodology): “Table A.8: Perplexity comparison of OPT models quantized into 4 and 3 bits using different methods including RTN, GPTQ, AWQ and SpQR on C4 and WikiText-2.”
OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models (2023, methodology): “Instead of directly learning a clipping threshold as did in previous methods (Esser et al. (2019); Choi et al.”
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (2024, unknown): “The majority of quantization schemes focus on compressing LLMs by using weight-only quantization , [Frantar et al., 2022, Dettmers et al., 2023, Lin et al., 2023, Egiazarian et al., 2024, Tseng et al., 2024].”
A Simple and Effective Pruning Approach for Large Language Models (2023, methodology): “This observation has spurred the development of various quantization schemes [14, 15, 38, 70] tailored specifically for LLMs to manage these outlier features.”
AI and Memory Wall (2024, unknown): “With current meth-ods, it is relatively easy to quantize inference down to INT4 precision, with minimal impact on accuracy [6, 9, 17, 23, 44].”