Q8BERT: Quantized 8Bit BERT

By Ofir Zafrir, Guy Boudoukh, Peter Izsak, Moshe Wasserblat. 2019

584 citations recorded by the source corpus.

Recently, pre-trained Transformer [1] based language models such as BERT [2] and GPT [3], have shown great improvement in many Natural Language Processing (NLP) tasks. However, these models contain a large amount of parameters. The emergence of even larger and more accurate models such as GPT2 [4] and Megatron11https://github.com/NVIDIA/Megatron-LM, suggest a trend of large pre-trained Transformer models. However, using these large models in production environments is a complex task requiring a large amount of compute, memory and power resources. In this work we show how to perform quantization-aware training during the fine-tuning phase of BERT in order to compress BERT by 4x with minimal accuracy loss. Furthermore, the produced quantized model can accelerate inference speed if it is optimized for 8bit Integer supporting hardware.

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, background): “…efficient have also been tackled from an extensive number of angles such as (but not limited to) distillation (Hinton et al., 2015), compression (Zafrir et al., 2019), parameter sharing (Lan et al., 2019; Tay et al., 2019; Zhang et al., 2021), efficient attention (Tay et al., 2020c; Kitaev et…”
Efficiently Scaling Transformer Inference (2022, neutral): “…(Sanh et al., 2019; Sun et al., 2020), and model compression techniques, such as pruning (Li et al., 2020b; Brix et al., 2020; Zhou et al., 2021; Li et al., 2020a; Wang et al., 2020), or quantization (Dettmers et al., 2022; Abdolrashidi et al., 2021; Zafrir et al., 2019; Zhang et al., 2018).”
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (2020, background): “While ongoing efforts in the NLU literature for distilling [15, 33], compressing [40], and pruning [20] BERT can be instrumental in narrowing this gap, they generally achieve significantly smaller speedups than our re-designed architecture for IR, due to their generic nature, and more aggressive…”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, background): “, 2020) and quantization (Gong et al., 2014; Zafrir et al., 2019; Stock et al., 2021) as well as early exit strategies for inference (Liu et al.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “, 2020), quantization (Zafrir et al., 2019; Shen et al., 2020; Tao et al., 2022), and pruning (Michel et al.”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “With this goal in mind, 8-bit quantization methods for transformers have been developed (Chen et al., 2020; Lin et al., 2020; Zafrir et al., 2019; Shen et al., 2020).”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, neutral): “[56, 76] are the ﬁrst few works to quantize BERT models using integer numbers for both weight and activations.”
The Right Tool for the Job: Matching Model and Instance Complexities (2020, methodology): “both speeds up numerical operations and reduces model size (Wróbel et al., 2018; Shen et al., 2019; Zafrir et al., 2019).”
A Primer in BERTology: What We Know About How BERT Works (2020, methodology): “The quantization approach aims to decrease BERT’s memory footprint through lowering the precision of its weights (Shen et al., 2019; Zafrir et al., 2019).”
Pre-Trained Models: Past, Present and Future (2021, methodology): “For recent Transformer-based models, 8-bit quantization has been proved to be effective for model compression in Q8BERT (Zafrir et al., 2019), with little impact on the model performance.”
Nyströmformer: A Nystöm-based Algorithm for Approximating Self-Attention (2021, background): “Weight pruning (Michel, Levy, and Neubig 2019), weight factorization (Lan et al. 2020), weight quantization (Zafrir et al. 2019) or knowledge distillation (Sanh et al. 2019) are several strategies that have been proposed to improve memory efﬁciency in transformers.”
Pre-trained models for natural language processing: A survey (2020, neutral): “Q-BERT [150] Quantization 12 HAWQ + GWQ BERTBASE ≈ 99% BERT Q8BERT [204] 12 DQ + QAT BERTBASE ≈ 99% BERT ALBERT§ [91] Param.”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, methodology): “We compare against Q8BERT (Zafrir et al., 2019) and Q-BERT (Shen et al.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “matrix multiplications and convolutions) are carried out with floating point arithmetic [15, 263].”
TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020, methodology): “In (Prato et al., 2019; Zafrir et al., 2019), 8-bit quantization is successfully applied to Transformer-based models with comparable performance as the full-precision baseline.”
Movement Pruning: Adaptive Sparsity by Fine-Tuning (2020, methodology): “Quantization has been applied to a variety of modern large architectures [Fan et al., 2020b, Zafrir et al., 2019, Gong et al., 2014] providing high memory compression rates at the cost of no or little performance.”
BinaryBERT: Pushing the Limit of BERT Quantization (2020, methodology): “Previous works on BERT quantization (Zafrir et al., 2019; Shen et al., 2020; Zhang et al., 2020) push down the bit-width to as low as two, but none of them achieves binarization.”
I-BERT: Integer-only BERT Quantization (2021, methodology): “, GELU, Softmax, and Layer Normalization [1]) are carried out with floating point arithmetic [52, 71].”