Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT

By Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zi-Mei Yao, A. Gholami, Michael W. Mahoney, Kurt Keutzer. 2019

675 citations recorded by the source corpus.

Transformer based architectures have become de-facto models used for a range of Natural Language Processing tasks. In particular, the BERT based models achieved significant accuracy gain for GLUE tasks, CoNLL-03 and SQuAD. However, BERT based models have a prohibitive memory footprint and latency. As a result, deploying BERT based models in resource constrained environments has become a challenging task. In this work, we perform an extensive analysis of fine-tuned BERT models using second order Hessian information, and we use our results to propose a novel method for quantizing BERT models to ultra low precision. In particular, we propose a new group-wise quantization scheme, and we use Hessian-based mix-precision method to compress the model further. We extensively test our proposed method on BERT downstream tasks of SST-2, MNLI, CoNLL-03, and SQuAD. We can achieve comparable performance to baseline with at most 2.3% performance degradation, even with ultra-low precision quantization down to 2 bits, corresponding up to 13× compression of the model parameters, and up to 4× compression of the embedding table as well as activations. Among all tasks, we observed the highest performance loss for BERT fine-tuned on SQuAD. By probing into the Hessian based analysis as well as visualization, we show that this is related to the fact that current training/fine-tuning strategy of BERT does not converge for SQuAD.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Efficiency & Compression

Citations in the corpus

Pretrained Transformers Improve Out-of-Distribution Robustness (2020, methodology): “This highlights that testing model compression methods for BERT (Shen et al., 2020; Ganesh et al., 2020; Li et al., 2020) on only in-distribution examples gives”
Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “Instead, we follow past work and report the memory needed to store the model parameters (Thakker et al., 2019; Shen et al., 2020).”
Stable and low-precision training for large-scale vision-language models (2023, background): “This not only decreases the model footprint and increases inference speed but adjusts the model to new data [2, 32, 75, 55, 76].”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “, 2020), quantization (Zafrir et al., 2019; Shen et al., 2020; Tao et al., 2022), and pruning (Michel et al.”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “With this goal in mind, 8-bit quantization methods for transformers have been developed (Chen et al., 2020; Lin et al., 2020; Zafrir et al., 2019; Shen et al., 2020).”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, methodology): “[56, 76] are the ﬁrst few works to quantize BERT models using integer numbers for both weight and activations.”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, background): “Satellited around BERT, there are also studies on model compression (Sun et al., 2019c; Jiao et al., 2019; Shen et al., 2019) and extension from understanding to generation (Chen et al., 2019a; Clinchant et al., 2019; Wang and Cho, 2019).”
Sparse is Enough in Scaling Transformers (2021, methodology): “Our approach is complementary to model quantization studies [35, 38, 28] that use fewer bits for the weights.”
The Right Tool for the Job: Matching Model and Instance Complexities (2020, methodology): “both speeds up numerical operations and reduces model size (Wróbel et al., 2018; Shen et al., 2019; Zafrir et al., 2019).”
Efficient Transformers: A Survey (2020, methodology): “Q-BERT [66] is a model that quantizes Transformer models to ultra-low precision.”
Pre-Trained Models: Past, Present and Future (2021, methodology): “Both Q-BERT and TernaryBERT result in ultra low-bit models.”
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020, methodology): “[29] quantized BERT to 2-bit using Hessian information.”
Pre-trained models for natural language processing: A survey (2020, neutral): “Q-BERT [143] Quantization 12 HAWQ + GWQ BERTBASE ≈ 99% BERT Q8BERT [196] 12 DQ + QAT BERTBASE ≈ 99% BERT ALBERT§ [86] Param.”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, methodology): “We compare against Q8BERT (Zafrir et al., 2019) and Q-BERT (Shen et al., 2020).”
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference (2020, methodology): “Q-BERT (Shen et al., 2019) uses quantization to compress BERT, and Layer-Drop (Fan et al., 2019) uses group regularization to enable structured pruning at inference time.”
Random Feature Attention (2021, background): “Note that there are plenty orthogonal choices for improving efﬁciency such as weight sharing (Dehghani et al., 2019), quantization (Shen et al., 2020), knowledge distillation (Sanh et al., 2020), and adapters (Houlsby et al., 2019).”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “For instance, this approach was found useful in Q-BERT [212] for quantizing Transformer [235] models that consist of fully-connected attention layers.”