AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration

By Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, Song Han. 2023

1,557 citations recorded by the source corpus.

Large language models (LLMs) have transformed numerous AI applications. On-device LLM is becoming increasingly important: running LLMs locally on edge devices can reduce cloud computing costs and protect users' privacy. However, the astronomical model size and the limited hardware resources pose significant deployment challenges. To solve these issues, we propose Activation-aware Weight Quantization (AWQ) and TinyChat, an algorithm-system full-stack solution for efficient on-device LLM deployment. AWQ is a novel quantization method that identifies and protects salient weights based on activation distribution, significantly reducing model size while preserving performance. TinyChat, an optimized inference framework, translates AWQ's theoretical memory savings into practical speedups through techniques such as on-the-fly dequantization, SIMD-aware weight packing, and kernel fusion. Together, they enable 4x model size reduction and 3-4x acceleration across various edge platforms, from high-end desktop GPUs to resource-constrained IoT devices. This solution democratizes on-device LLM deployment, offering privacy-preserving, low-latency AI capabilities across a wide range of applications.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Large Language Models, Efficiency & Compression, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Large Language Models

In Efficiency & Compression

In Signal Processing

Citations in the corpus

Fast Inference of Mixture-of-Experts Language Models with Offloading (2023, critical): “While there are more extreme algorithms for 3-and even 2-bit compression Chee et al. (2023); Lin et al. (2023); Dettmers et al. (2023), they are typically inferior to choosing a smaller model and quantizing it to around 4 bits.”
Seed-TTS: A Family of High-Quality Versatile Speech Generation Models (2024, methodology): “To resolve these challenges, we employed various techniques to reduce inference cost and latency [Dao et al., 2022, Ainslie et al., 2023, Luo et al., 2023, Lin et al., 2023].”
OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models (2023, methodology): “As depicted in Figure 4, we compare RTN, AWQ (Lin et al. (2023)), and our proposed OmniQuant.”
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (2024, background): “The trend has been to move from 16 bits to lower bits, such as 4-bit variants [FAHA23, LTT + 23].”
SqueezeLLM: Dense-and-Sparse Quantization (2023, methodology): “This indeed results in the activation-based sensitivity/importance metric as in the AWQ framework [32].”
Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models (2025, neutral): “Unlike model compression techniques such as quantization [29,50,97] or KV cache compression [55, 76, 107, 110], which focus on reducing model size for lightweight inference, efficient reasoning in LLMs emphasizes smart and concise reasoning by optimizing the length of generated reasoning sequences…”
BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains (2024, background): “On the other hand, MedMCQA experiences a notable average performance drop of 4.05% across all quantization methods, while PubMedQA shows a remarkable 24.1% increase in accuracy when employing the AWQ method.”
Harnessing the Wisdom of LLM Crowds through Complementarity-Driven Iterative Collaboration (2026, unknown): “4 All models are quantized to 4-bit precision (i.e., Q4_K_M scheme) to improve memory efficiency and support practical multi-model deployment (Lin et al. 2024b).”
When Is 0.1% Enough? Analyzing the Combined Effects of Dimensionality Reduction and Quantization on Text Embedding Compression (2026, unknown): “Quantization is widely used for neural model compression, including reduced precision weights and activations for BERT and LLMs (Zafrir et al., 2019; Frantar et al., 2022; Xiao et al., 2023; Lin et al., 2024).”