GPT3.int8(): 8-bit Matrix Multiplication for Transformers at Scale

By Tim Dettmers, M. Lewis, Younes Belkada, Luke S. Zettlemoyer. Neural Information Processing Systems, 2022

933 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/b8b45b14df9029562b8995c6ab7fd90a8810f312

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Leveraging large language models for predictive chemistry (2024, unknown): “We also performed some of our experiments by fine-tuning the GPT-J-6B model 52,53 (which has been trained on the Pile dataset 54 ) on consumer hardware using 8-bit quantization 55 and 8-bit optimizers 56 in addition to the low-rank adaptation (LoRA) technique 57 .”
Big Little Transformer Decoder (2023, methodology): “Well-known approaches include efficient architecture design [25, 31, 34, 57, 64, 72], quantization [8, 29, 51, 71, 76, 78, 79], pruning [11, 13, 32, 33, 39, 46, 61], and neural architecture search [5, 53, 54, 63, 73, 77].”
The Platonic Representation Hypothesis (2024, background): “As transformer architectures have “emergent outliers” (Dettmers et al., 2022), we truncate the elements in the features that are above the 95 -th percentile.”
LLaVA-Prumerge: Adaptive Token Reduction for Efficient Large Multimodal Models (2024, unknown): “5 [28] employ advanced compression techniques such as 4/8 bit quantization [11, 38].”
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models (2024, unknown): “Various quantization strategies (Dettmers et al., 2022a; Frantar et al., 2023; Lin et al., 2023; Egiazarian et al., 2024) decrease memory requirements in large language models (LLMs) by utilizing lower-precision representations for weights.”
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (2024, unknown): “Several works show that, unlike weights, quantizing the activations is hard due to the outlier features [Wei et al., 2022, Dettmers et al., 2022, Xiao et al., 2023].”
OpenVLA: An Open-Source Vision-Language-Action Model (2024, methodology): “In this section, we test whether we can further reduce the required memory for policy inference and broaden accessibility of VLA policies, by using modern quantization techniques developed for serving LLMs [27, 88].”
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (2023, methodology): “LLaMA-2-70B–ran using 4-bit quantization (Dettmers et al., 2022)–yields a median spread of 0.”
SqueezeLLM: Dense-and-Sparse Quantization (2023, methodology): “Notably, [10] keeps outlier activations in floating-point, while [46] transfers outlier factors to later layers without affecting functionality.”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “In the realm of Model Compression, Quantization (Frantar et al., 2023; Dettmers et al., 2022; Kim et al., 2023c; Tao et al., 2022b; Yao et al., 2022; Xiao et al., 2023), Parameter Pruning (Ma et al., 2023d; Zhang et al., 2023h; Frantar and Alistarh, 2023), and Low-Rank Approximation (Xu et al.,…”
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (2023, unknown): “Additionally, alternative approaches explore quantization methods specifically tailored to LLMs, as discussed in (Frantar et al., 2022; Xiao et al., 2022; Yao et al., 2022; Dettmers et al., 2022b;a).”
Fine-Tuning Language Models with Just Forward Passes (2023, methodology): “We did not explore combining MeZO with other memory-efficient methods, such as FlashAttention [23] and quantization [26], though we hope to investigate this in the future.”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, background): “In response to this, some strategies have resorted to quantizing the KV cache [159], [36], but quantization will certainly compromises performance.”
RAS: Retrieval-And-Structuring for Knowledge-Intensive LLM Generation (2025, unknown): “Prob.”