OPTQ: Accurate Quantization for Generative Pre-trained Transformers

By Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh. International Conference on Learning Representations, 2023

431 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (2024, methodology): “One approach to address these challenges is to use post-training quantization to create low-bit models for inference [XLS + 23, FAHA23, CCKS23, TCS + 24].”
AI and Memory Wall (2024, unknown): “With current meth-ods, it is relatively easy to quantize inference down to INT4 precision, with minimal impact on accuracy [6, 9, 17, 23, 44].”
QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks (2024, methodology): “PTQ requires less compute than QAT and achieves competitive performance (Chee et al., 2023; Frantar et al., 2023; Shao et al., 2024; Egiazarian et al., 2024).”
QuIP: 2-Bit Quantization of Large Language Models With Guarantees (2023, methodology): “This work studies the post-training quantization of LLM parameters as a way to improve their runtime efficiency [4, 8, 22, 31, 33, 34].”