LLM-QAT: Data-Free Quantization Aware Training for Large Language Models

By Zechun Liu, Barlas Oğuz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, Vikas Chandra. Annual Meeting of the Association for Computational Linguistics, 2023

398 citations recorded by the source corpus.

Several post-training quantization methods have been applied to large language models (LLMs), and have been shown to perform well down to 8-bits. We find that these methods break down at lower bit precision, and investigate quantization aware training for LLMs (LLM-QAT) to push quantization levels even further. We propose a data-free distillation method that leverages generations produced by the pre-trained model, which better preserves the original output distribution and allows quantizing any generative model independent of its training data, similar to post-training quantization methods. In addition to quantizing weights and activations, we also quantize the KV cache, which is critical for increasing throughput and support long sequence dependencies at current model sizes. We experiment with LLaMA models of sizes 7B, 13B, and 30B, at quantization levels down to 4-bits. We observe large improvements over training-free methods, especially in the low-bit settings.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/6bd3ee1ca608bc66a490f63f2fb107d79b44f3e2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Kurt Keutzer
2. Tijmen Blankevoort
3. A. Gholami
4. Sehoon Kim
5. Yu Qiao
6. Coleman Hooper
7. Ping Luo
8. Peng Gao

Citations in the corpus

• A Comprehensive Overview of Large Language Models (2023, neutral): “Following this method, LLMs can be easily and effectively extended as multimodal chatbots [208], [202], [210] and multimodal task solvers [211], [212], [213].”
• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models (2023, methodology): “For instance, GPTQ (Frantar et al., 2022), a PTQ approach, can complete the quantization of LLaMA-13B in an hour using 128 samples on a single A100 GPU, while LLM-QAT (Liu et al., 2023b) requires 100k samples and hundreds of GPU hours.”
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (2024, background): “Other works quantized KV cache activations to 4-bits but required re-training to maintain performance [21].”
• SpinQuant: LLM quantization with learned rotations (2024, methodology): “This represents a mere 2.9 point gap from the full-precision network, a substantial improvement over the previous LLM-QAT [26] approach, which exhibited a 22.0 point gap under identical precision conditions.”