QuIP: 2-Bit Quantization of Large Language Models With Guarantees

By Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, Chris De Sa. Neural Information Processing Systems, 2023

433 citations recorded by the source corpus.

This work studies post-training parameter quantization in large language models (LLMs). We introduce quantization with incoherence processing (QuIP), a new method based on the insight that quantization benefits from incoherent weight and Hessian matrices, i.e., from the weights being even in magnitude and the directions in which it is important to round them accurately being unaligned with the coordinate axes. QuIP consists of two steps: (1) an adaptive rounding procedure minimizing a quadratic proxy objective; (2) efficient pre- and post-processing that ensures weight and Hessian incoherence via multiplication by random orthogonal matrices. We complement QuIP with the first theoretical analysis for an LLM-scale quantization algorithm, and show that our theory also applies to an existing method, OPTQ. Empirically, we find that our incoherence preprocessing improves several existing quantization algorithms and yields the first LLM quantization methods that produce viable results using only two bits per weight. Our code can be found at https://github.com/Cornell-RelaxML/QuIP.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/56b828717f32251a5e0f0be9c0113077f23c8429

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Furu Wei
2. Shaohan Huang
3. Shuming Ma
4. Yu Qiao
5. Wenhui Wang
6. Ping Luo
7. Dan Alistarh
8. Elias Frantar

Citations in the corpus

• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models (2023, background): “1 (Chee et al., 2023) restore the capabilities of the quantized model through parameter-efficient fine-tuning.”
• QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models (2023, background): “The next significant challenge is that existing large-scale quantization implementations, in particular for GPTQ and related methods (Frantar et al., 2022; Chee et al., 2023), are designed to be fast and memory efficient for the massive individual layers occurring in dense models.”
• SpinQuant: LLM quantization with learned rotations (2024, neutral): “Detailed GPU latency results are provided in the Appendix.”
• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (2024, methodology): “One approach to address these challenges is to use post-training quantization to create low-bit models for inference [XLS + 23, FAHA23, CCKS23, TCS + 24].”
• QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks (2024, methodology): “In QuIP (Chee et al., 2023), the 2-factor Kronecker approach achieves µ KronW = A 2 log (4 Cmn/δ ) 2 , where A and C are global constants independent of n and the number of factors.”
• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024, positive): “QuIP [8] and QuIP#[57] use incoherent processing to reduce the quantization, and we adapted this technique for FP8 F LASH A TTENTION -3.”
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (2024, neutral): “The idea of incoherence processing was introduced by [Chee et al., 2024] in the context of weight normalization for weight-only LLM quantization.”