BinaryBERT: Pushing the Limit of BERT Quantization

By Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael R. Lyu, Irwin King. 2020

276 citations recorded by the source corpus.

The rapid development of large pre-trained language models has greatly increased the demand for model compression techniques, among which quantization is a popular solution. In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization. We find that a binary BERT is hard to be trained directly than a ternary counterpart due to its complex and irregular loss landscape. Therefore, we propose ternary weight splitting, which initializes BinaryBERT by equivalently splitting from a half-sized ternary network. The binary model thus inherits the good performance of the ternary one, and can be further enhanced by fine-tuning the new architecture after splitting. Empirical results show that our BinaryBERT has only a slight performance drop compared with the full-precision model while being 24x smaller, achieving the state-of-the-art compression results on the GLUE and SQuAD benchmarks. Code will be released.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Efficiency & Compression

Citations in the corpus

AdapterDrop: On the Efficiency of Adapters in Transformers (2020, background): “(1) Smaller and faster models that are either distilled or trained from scratch (Sanh et al., 2019; Sun et al., 2020; Bai et al., 2021; Wang et al., 2020).”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “Versions of 8-bit BERT/RoBERTa include Q8BERT (Zafrir et al., 2019), QBERT (Shen et al., 2020), product quantization with quantization noise (Fan et al., 2020), TernaryBERT (Zhang et al., 2020), and BinaryBERT (Bai et al., 2021).”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “This analysis results also indicate why more expensive hidden-states knowledge distillation [2, 36] is used for ultra-low precision quantization to close the accuracy gap.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Recently, there have been attempts to extend this idea to Natural Language Processing (NLP) tasks [7, 119, 270].”
Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models (2022, unknown): “[23, 24] combine distillation with quantization.”
Intriguing Properties of Compression on Multilingual Models (2022, background): “…techniques have minimal impact on aggregate performance numbers (Gale et al., 2019; Li et al., 2020; Hou et al., 2020; Chen et al., 2021; Bai et al., 2020; ab Tessera et al., 2021), the impact on individual sub-populations in the data, such as low-resource languages, can be far more severe…”
Adaptable Adapters (2022, methodology): “…but often require re-training or fine-tuning the smaller model (Tang et al., 2019; Sanh et al., 2019; Voita et al., 2019; Sun et al., 2020; Bai et al., 2021), and (b) ondemand network size reduction at the inference time.2 There are two different approaches in the second category, namely…”
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (2023, unknown): “…recent works have explored language model compression, they are mostly focused on smaller models (Zafrir et al., 2019; Fan et al., 2020; Shen et al., 2020b; Zadeh et al., 2020; Bai et al., 2021; Qin et al., 2021; Liu et al., 2022b) like BERT (Devlin et al., 2019) or BART (Lewis et al., 2019).”
Sharpness-aware Quantization for Deep Neural Networks (2021, methodology): “Existing quantization methods can be roughly divided into two categories according to the quantization bitwidth, namely, fixed-point quantization [4,7,9,19,22,25,28,32,46,72,75,77] and binary quantization [1, 26, 41, 45, 47, 59, 60].”
A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development (2023, methodology): “Quantization-based approaches [120], [121] use 8-bit or even binary numbers to store the model weights, thereby reducing the model size compared to 32-bit storage.”
Learned Token Pruning for Transformers (2021, background): “These can be broadly categorized as follows: (i) efficient architecture design [8, 21, 24, 26, 27, 36, 45, 47, 50, 60]; (ii) knowledge distillation [23, 38, 42–44]; (iii) quantization [2, 4, 13, 15, 25, 40, 58, 59, 61]; and (iv) pruning.”
Extreme Compression for Pre-trained Transformers Made Simple and Efficient (2022, methodology): “There are prior works that show the feasibility of using only ternary or even binary weights [61, 3].”
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation (2021, background): “To-date, evaluating the merits and trade-offs incurred by compression have overwhelmingly centered on settings where the data is relatively abundant (Gale et al., 2019; Li et al., 2020a; Hou et al., 2020; Chen et al., 2021; Bai et al., 2020; ab Tessera et al., 2021).”
Post-Training Quantization for Vision Transformer (2021, background): “[1] further push BERT quantization to the limit with weight binarization.”
Confident Adaptive Language Modeling (2022, methodology): “Improving inference-time efficiency of LLMs has been an ongoing effort of the research community over the past several years [49; 72; 85], leveraging techniques such as knowledge distillation [6; 32; 36; 69; 69; 78; 56], floating point quantization [71; 65], layer pruning [24], vector dropping [38],”
Compression of Generative Pre-trained Language Models via Quantization (2022, methodology): “As the bit-width further decreases to 2, both LSQ and PACT fail on all datasets, despite their good performance on understanding tasks on BERT (Bai et al., 2021).”
SqueezeLLM: Dense-and-Sparse Quantization (2023, methodology): “Quantization-Aware Training (QAT) requires retraining the model to adapt its weights to help recover accuracy after quantization [1,27,39, 51,53,54], whereas Post-Training Quantization (PTQ) does not involve retraining [4,31,35,40,55].”
MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Models (2021, background): “These efforts can be generally categorized as follows: (i) knowledge distillation [21, 41, 35, 39]; (ii) quantization [3, 54, 37, 12, 53, 55, 2, 10]; (iii) new architecture design [40, 19, 23, 22, 45]; and (iv) pruning.”