OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models

By Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqiang Li, Kaipeng Zhang, Peng Gao, Yu Qiao, Ping Luo. International Conference on Learning Representations, 2023

460 citations recorded by the source corpus.

Large language models (LLMs) have revolutionized natural language processing tasks. However, their practical deployment is hindered by their immense memory and computation requirements. Although recent post-training quantization (PTQ) methods are effective in reducing memory footprint and improving the computational efficiency of LLM, they hand-craft quantization parameters, leading to low performance, especially in extremely low-bit quantization. To tackle this issue, we introduce an Omnidirectionally calibrated Quantization (\textbf{OmniQuant}) technique for LLMs, which achieves good performance in diverse quantization settings while maintaining the computational efficiency of PTQ by efficiently optimizing various quantization parameters. OmniQuant comprises two innovative components including Learnable Weight Clipping (LWC) and Learnable Equivalent Transformation (LET). LWC modulates the extreme values of weights by optimizing the clipping threshold. Meanwhile, LET tackles activation outliers by shifting the challenge of quantization from activations to weights. Operating within a differentiable framework using block-wise error minimization, OmniQuant can optimize the quantization process efficiently for both weight-only and weight-activation quantization. For instance, the LLaMA-2 model family size 7-70B can be processed with OmniQuant on a single A100-40G GPU within 1-16 hours using 128 samples. Extensive experiments validate OmniQuant's superior performance across diverse quantization configurations such as W4A4 (4-bit weight, 4-bit activation), W6A6, W4A16, W3A16, and W2A16. Additionally, OmniQuant demonstrates effectiveness in instruction-tuned models and delivers notable improvements in inference speed and memory reduction on real devices. Codes are available at \url{https://github.com/OpenGVLab/OmniQuant}.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/eb2c2330177f765038a2b17e2ee3498965865797

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

SpinQuant: LLM quantization with learned rotations (2024, methodology): “Noteworthily, SpinQuant no had W4A8 quantization achieves comparable performance as state-of-the-art weight-only quantization methods like QuIP# [42] and OminiQuant [37] on LLaMA-2.”
QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks (2024, methodology): “For example, AWQ (Lin et al., 2023) scales model weights by information from activations and OmniQuant (Shao et al., 2024) uses simple learnable model-preserving transformations.”
SqueezeLLM: Dense-and-Sparse Quantization (2023, unknown): “We see a similar trend as the zero-shot MMLU evaluation results where SqueezeLLM consistently outperforms the baseline quantization methods with the same model size.”
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (2024, unknown): “We compare against 4-bit SmoothQuant [Xiao et al., 2023] and OmniQuant [Shao et al., 2023].”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “including neural network pruning ([19], [20]), low-rank factor-izations of the weight matrices ([21], [22]), quantization ([23], [24]), knowledge distillation ([25], [26]), neural architecture search ([27], [28]), and other techniques ([29], [30]).”