QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
By Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, Christopher De Sa. International Conference on Machine Learning, 2024
366 citations recorded by the source corpus.
Post-training quantization (PTQ) reduces the memory footprint of LLMs by quantizing their weights to low-precision. In this work, we introduce QuIP#, a weight-only PTQ method that achieves state-of-the-art results in extreme compression regimes (≤ 4 bits per weight) using three novel techniques. First, QuIP# improves QuIP’s (Chee et al., 2023) incoherence processing by using the randomized Hadamard transform, which is faster and has better theoretical properties. Second, QuIP# uses vector quantization to take advantage of the ball-shaped sub-Gaussian distribution that incoherent weights possess: specifically, we introduce a set of hardware-efficient codebooks based on the highly symmetric E8 lattice, which achieves the optimal 8-dimension unit ball packing. Third, QuIP# uses fine-tuning to improve fidelity to the original model. Our experiments show that QuIP# outperforms existing PTQ methods, enables new behaviors in PTQ scaling, and supports fast inference. Our code can be found at https://github.com/Cornell-RelaxML/quip-sharp.
View Paper: https://www.semanticscholar.org/paper/8fbf2eb4587b5c271979c3f96eee1b109496143e
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Tijmen Blankevoort
- Zechun Liu
- Changsheng Zhao
- Igor Fedorov
- Bilge Soran
- Dhruv Choudhary
- Vikas Chandra
- Raghuraman Krishnamoorthi
Citations in the corpus
- SpinQuant: LLM quantization with learned rotations (2024, methodology): “Random Hadamard matrices 2 outperform random rotation matrices, in consistent with the findings in [42] that Hadamard matrices yield tighter bounds on weight maximal value.”
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024, unknown): “[8]andTsengetal.”
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (2024, unknown): “Two weight-only quantization methods, QuIP [Chee et al., 2024] and QuIP# [Tseng et al., 2024] have previously considered improving quantization by applying rotations.”