SqueezeLLM: Dense-and-Sparse Quantization

By Sehoon Kim, Coleman Hooper, A. Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, Kurt Keutzer. International Conference on Machine Learning, 2023

345 citations recorded by the source corpus.

Generative Large Language Models (LLMs) have demonstrated remarkable results for a wide range of tasks. However, deploying these models for inference has been a significant challenge due to their unprecedented resource requirements. This has forced existing deployment frameworks to use multi-GPU inference pipelines, which are often complex and costly, or to use smaller and less performant models. In this work, we demonstrate that the main bottleneck for generative inference with LLMs is memory bandwidth, rather than compute, specifically for single batch inference. While quantization has emerged as a promising solution by representing weights with reduced precision, previous efforts have often resulted in notable performance degradation. To address this, we introduce SqueezeLLM, a post-training quantization framework that not only enables lossless compression to ultra-low precisions of up to 3-bit, but also achieves higher quantization performance under the same memory constraint. Our framework incorporates two novel ideas: (i) sensitivity-based non-uniform quantization, which searches for the optimal bit precision assignment based on second-order information; and (ii) the Dense-and-Sparse decomposition that stores outliers and sensitive weight values in an efficient sparse format. When applied to the LLaMA models, our 3-bit quantization significantly reduces the perplexity gap from the FP16 baseline by up to 2.1x as compared to the state-of-the-art methods with the same memory requirement. Furthermore, when deployed on an A6000 GPU, our quantized models achieve up to 2.3x speedup compared to the baseline. Our code is available at https://github.com/SqueezeAILab/SqueezeLLM.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/3b7ef6f9f27e33e6a4e3bfac90dcb01ab09718bc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

SpinQuant: LLM quantization with learned rotations (2024, neutral): “A Appendix / supplemental material A.1”
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (2024, methodology): “Among them, quantization techniques have been widely applied (Frantar et al., 2022; Lin et al., 2023; Kim et al., 2023; Xu et al., 2023).”
QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks (2024, positive): “Compared to existing “fast inference” quantization meth-ods such as SpQR (Dettmers et al., 2023) and SqueezeLLM (Kim et al., 2023), we again find that QuIP # offers significantly higher throughput ( > 40% ) at the same or better quantization quality.”
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (2023, neutral): “However, the question practitioner willing to deploy a model for a specific application would ask is: What is the best model and compression setup for a given memory constraint?”
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (2024, methodology): “By allowing for more flexible quantization signpost placement, relative to uniform quantization methods, this enables improved accuracy for the same bit precision [6, 16].”
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (2024, background): “Kim et al. (2023) introduce a sparse plus low-precision pattern to handle a minor portion of vital weights, among other techniques.”