GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference

By Ali Hadi Zadeh, Andreas Moshovos. 2020

230 citations recorded by the source corpus.

Attention-based models have demonstrated remarkable success in various natural language understanding tasks. However, efficient execution remains a challenge for these models which are memory-bound due to their massive number of parameters. We present GOBO, a model quantization technique that compresses the vast majority (typically 99.9%) of the 32-bit floating-point parameters of state-of-the-art BERT models and their variants to 3 bits while maintaining their accuracy. Unlike other quantization methods, GOBO does not require fine-tuning nor retraining to compensate for the quantization error. We present two practical hardware applications of GOBO. In the first GOBO reduces memory storage and traffic and as a result inference latency and energy consumption. This GOBO memory compression mechanism is plug-in compatible with many architectures; we demonstrate it with the TPU, Eyeriss, and an architecture using Tensor Cores-like units. Second, we present a co-designed hardware architecture that also reduces computation. Uniquely, the GOBO architecture maintains most of the weights in 3b even during computation, a property that: (i) makes the processing elements area efficient, allowing us to pack more compute power per unit area, (ii) replaces most multiply-accumulations with additions, and (iii) reduces the off-chip traffic by amplifying on-chip memory capacity.

Subfields: Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, neutral): “Along this line, one of the first works applied to Transformer-based [64] models is [75].”
A Primer in BERTology: What We Know About How BERT Works (2020, neutral): “, 2019) ×4 99% − BERT12 No WNLI, MNLI; SQuAD GOBO (Zadeh and Moshovos, 2020) ×9.”
TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020, methodology): “Speciﬁcally, In Q-BERT (Shen et al., 2020) and GOBO (Zadeh and Moshovos, 2020), mixed-precision with 3 or more bits are used to avoid severe accuracy drop.”
BinaryBERT: Pushing the Limit of BERT Quantization (2020, background): “Therefore, weight binarization may severely harm the performance, which may explain why most current approaches stop at 2-bit weight quantization (Shen et al., 2020; Zadeh and Moshovos, 2020; Zhang et al., 2020).”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “To reduce performance drop in ultra-low (1 or 2) bit models, researchers proposed methods like mixed-bit quantization [177], [179], combining knowledge distillation with quantization [178], and product quantization (PQ) [180].”
EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (2024, methodology): “…models, involving techniques such as distillation (Hin-ton et al., 2015), quantization (Hubara et al., 2018; Shen et al., 2020; Kim et al., 2021; Zadeh et al., 2020; Zafrir et al., 2019), pruning (Gale et al., 2019; Sanh et al., 2020; Kur-tic et al., 2022; Voita et al., 2019), and innovative…”
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (2023, unknown): “For instance, over 5 million inferences, a full precision 30B model would take 4280 A100 GPU days, while PTQ 8-8-8 would take 2743 days, and a comparable accuracy 4-8-8 LLM-QAT model would take 586 days 1 .”
DKM: Differentiable K-Means Clustering Layer for Neural Network Compression (2021, methodology): “Base GOBO [29] DKM DKM Metrics 32bit xform†3,emb‡4 xform4/2§,emb4 xform5/2,emb3”
MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Models (2021, background): “These efforts can be generally categorized as follows: (i) knowledge distillation [21, 41, 35, 39]; (ii) quantization [3, 54, 37, 12, 53, 55, 2, 10]; (iii) new architecture design [40, 19, 23, 22, 45]; and (iv) pruning.”
Big Little Transformer Decoder (2023, methodology): “Well-known approaches include efficient architecture design [25, 31, 34, 57, 64, 72], quantization [8, 29, 51, 71, 76, 78, 79], pruning [11, 13, 32, 33, 39, 46, 61], and neural architecture search [5, 53, 54, 63, 73, 77].”
KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization (2021, methodology): “set Q8BERT Zafrir et al. (2019) and TernaryBERT Zhang et al.”
Utility Is in the Eye of the User: A Critique of NLP Leaderboard Design (2020, background): “There are many others like it as well (Zadeh and Moshovos, 2020; Hou et al., 2020; Mao et al., 2020).”
Extreme Compression for Pre-trained Transformers Made Simple and Efficient (2022, background): “[59] Ali Hadi Zadeh, Isak Edo, Omar Mohamed Awad, and Andreas Moshovos.”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “To alleviate the issue of generating unsatisfactory text with truncated PLMs, a promising solution is to first identify important weights and then avoid truncating them during the quantization step [204].”
Compression of Generative Pre-trained Language Models via Quantization (2022, background): “There have been many attempts to quantize task-speciﬁc BERT models (Zafrir et al., 2019; Shen et al., 2020; Zadeh et al., 2020) with only negligible performance drop on natural language understanding tasks.”
Learned Token Pruning for Transformers (2021, background): “First, due to the binary nature of 𝑀 there is no gradient flow for pruned tokens.”