AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

By Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, Song Han. arXiv.org, 2023

857 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/2b7c9fd2a94deaee3e7e56dc57bab0bd39d3683c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

A Simple and Effective Pruning Approach for Large Language Models (2023, methodology): “This observation has spurred the development of various quantization schemes (Dettmers et al., 2022; Xiao et al., 2023; Lin et al., 2023; Dettmers et al., 2023; Behdin et al., 2023) tailored specifically for LLMs to handle outlier features.”
Foundation models in robotics: Applications, challenges, and the future (2023, neutral): “Some recent work has attempted this approach (though without an explicit link to robotics) [215].”
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (2023, neutral): “Additionally, alternative approaches explore quantization methods specifically tailored to LLMs, as discussed in [49, 50, 51, 52, 53, 54].”
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models (2024, positive): “Various quantization strategies (Dettmers et al., 2022a; Frantar et al., 2023; Lin et al., 2023; Egiazarian et al., 2024) decrease memory requirements in large language models (LLMs) by utilizing lower-precision representations for weights.”
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (2024, methodology): “Our analysis suggests: • For key cache, there are a few fixed channels whose magnitudes are very large, which is consistent with previous finding (Lin et al., 2023; Xiao et al., 2023a).”
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (2024, neutral): “The majority of quantization schemes focus on compressing LLMs by using weight-only quantization , [Frantar et al., 2022, Dettmers et al., 2023, Lin et al., 2023, Egiazarian et al., 2024, Tseng et al., 2024].”
SpinQuant: LLM quantization with learned rotations (2024, methodology): “Prior research mitigates this challenge by trading quantization difficulty between weights and activations [46, 23] or employing mixed-precision to handle outliers [50].”
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (2023, neutral): “G Choice of optimal LLM configuration for specific hardware In the preceding discussion, we were searching for optimal model configuration given some compression target without targeting any specific hardware or device.”
SGLang: Efficient Execution of Structured Language Model Programs (2023, neutral): “Additionally, SGLang is compatible with many other common inference optimizations [60, 39, 3, 23, 59, 10, 26, 15, 19, 32, 31, 11].”
AI and Memory Wall (2024, positive): “With current meth-ods, it is relatively easy to quantize inference down to INT4 precision, with minimal impact on accuracy [6, 9, 17, 23, 44].”