nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models

By Gunho Park, Baeseong Park, S. Kwon, Byeongwook Kim, Youngjoo Lee, Dongsoo Lee. 2022

77 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Large Language Models

In Efficiency & Compression

Citations in the corpus

QLoRA: Efficient Finetuning of Quantized LLMs (2023, background): “[67] and LLM.int8() [14]) while others use more sophisticated grouping methods [44, 70].”
The case for 4-bit precision: k-bit Inference Scaling Laws (2022, methodology): “Compared to smaller models, LLM quantization poses some unique challenges, such as emergent outliers (Dettmers et al., 2022a; Zeng et al., 2022; Xiao et al., 2022) and optimized low-bit inference for LLMs (Frantar et al., 2022; Park et al., 2022; Yao et al., 2022).”
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (2022, methodology): “, 2022) and nuQmm (Park et al., 2022) use a per-tensor and group-wise quantization scheme for LLMs, which requires customized CUDA kernels.”
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022, methodology): “For example, it is easily combined with standard grouping (Alistarh et al., 2017; Park et al., 2022), i.”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “There are two methods that were developed in parallel to ours: nuQmm (Park et al., 2022) and ZeroQuant (Yao et al., 2022).”
AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration (2023, unknown): “…to INT8 (Dettmers et al., 2022; Xiao et al., 2022; Yao et al., 2022; Wei et al., 2022a; 2023); (2) Low-bit weight-only quantization ( e.g ., W4A16), where only weights are quantized into low-bit integers (Frantar et al., 2022; Dettmers & Zettlemoyer, 2022; Sheng et al., 2023; Park et al., 2022).”
High-throughput Generative Inference of Large Language Models with a Single GPU (2023, methodology): “Both spar-sification (Hoefler et al., 2021; Frantar & Alistarh, 2023) and quantization (Kwon et al., 2022; Yao et al., 2022; Park et al., 2022; Xiao et al., 2022; Frantar et al., 2022; Dettmers et al., 2022) have been adopted for LLM inference.”
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect (2024, methodology): “MiniLLM Gu et al. (2023) uses reverse Kullback-Leibler divergence to prevent the student model from overestimating the low-probability regions of the teacher distribution.”
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (2023, unknown): “Due to computational constraints, early work such as ZeroQuant [YAZ + 22], LLM.int8() [DLBZ22], and nuQmm [PPK + 22] used direct rounding of weights to the nearest quantization level, while customizing the quantization granularity (i.e., group size) to trade off space for increased accuracy.”
QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models (2023, methodology): “We consider an individual user application, like (Frantar et al., 2022; Leviathan et al., 2023; Park et al., 2022), where a single prompt (sampled from C4) should be processed to generate a 128-token response.”
A Survey on Model Compression for Large Language Models (2023, unknown): “Precision Methods 8-bit quantization LUT-GEMM [Park et al., 2022], LLM.”
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (2023, methodology): “, W4A16), where only weights are quantized into low-bit integers [17, 13, 44, 39].”
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time (2023, methodology): “For example, zeroQuant (Yao et al., 2022) and nuQmm (Park et al., 2022) implement customized CUDA kernels to support tenor-wise or group-wise quantization for LLM inference; LLM.int8 (Dettmers et al., 2022) adopts a mixed INT8/FP16 computation to diminish the influence of activation outliers;…”
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time (2023, background): “[9] Gunho Park, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee.”
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (2023, methodology): “In terms of datasets and evaluation, we mainly focus on perplexity on the raw WikiText2 test set [31], a popular benchmark in LLM compression literature [47, 36, 9, 46].”