TernaryBERT: Distillation-aware Ultra-low Bit BERT

By Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, Qun Liu. 2020

238 citations recorded by the source corpus.

Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks.However, these models are both computation and memory expensive, hindering their deployment to resource-constrained devices. In this work, we propose TernaryBERT, which ternarizes the weights in a fine-tuned BERT model. Specifically, we use both approximation-based and loss-aware ternarization methods and empirically investigate the ternarization granularity of different parts of BERT. Moreover, to reduce the accuracy degradation caused by the lower capacity of low bits, we leverage the knowledge distillation technique in the training process. Experiments on the GLUE benchmark and SQuAD show that our proposed TernaryBERT outperforms the other BERT quantization methods, and even achieves comparable performance as the full-precision model while being 14.9x smaller.

Subfields: Electrical and Electronic Engineering, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electrical and Electronic Engineering

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Stable and low-precision training for large-scale vision-language models (2023, background): “This not only decreases the model footprint and increases inference speed but adjusts the model to new data [2, 32, 75, 55, 76].”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “, 2020), TernaryBERT (Zhang et al., 2020), and BinaryBERT (Bai et al.”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Furthermore, in order to achieve better accuracy performance, hidden-states knowledge distillation, e.g., [2, 79], is usually applied for the quantized model.”
Pre-Trained Models: Past, Present and Future (2021, methodology): “TernaryBERT (Zhang et al., 2020b) applies knowledge distillation in quantization, forcing low-bit models to imitate full-precision models.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Recently, there have been attempts to extend this idea to Natural Language Processing (NLP) tasks [7, 119, 270].”
I-BERT: Integer-only BERT Quantization (2021, methodology): “While (Bai et al., 2020; Bhandare et al., 2019; Zafrir et al., 2019; Zhang et al., 2020) attempt to process Embedding and MatMul efficiently with integer arithmetic, they keep the remaining operations (i.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “To reduce performance drop in ultra-low (1 or 2) bit models, researchers proposed methods like mixed-bit quantization [177], [179], combining knowledge distillation with quantization [178], and product quantization (PQ) [180].”
Learned Token Pruning for Transformers (2021, background): “These can be broadly categorized as follows: (i) efficient architecture design [8, 21, 24, 26, 27, 36, 45, 47, 50, 60]; (ii) knowledge distillation [23, 38, 42–44]; (iii) quantization [2, 4, 13, 15, 25, 40, 58, 59, 61]; and (iv) pruning.”
Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models (2022, background): “[ 23, 24] combine distillation with quantization.”
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (2023, unknown): “Thus, many recent work (Shen et al., 2020a; Zhang et al., 2020) adopt clipping-based quantization for transformer-based language models.”
Compression of Generative Pre-trained Language Models via Quantization (2022, methodology): “In practice, following previous works on BERT quantization (Zhang et al., 2020; Bai et al., 2021), we use layer-wise quantization ( i.e. , one clipping factor for elements in each weight matrix) for all weight matrices in the Transformer layers and row-wise quantization ( i.e. , one clipping factor…”
Extreme Compression for Pre-trained Transformers Made Simple and Efficient (2022, methodology): “There are prior works that show the feasibility of using only ternary or even binary weights [61, 3].”
MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Models (2021, background): “These eﬀorts can be generally categorized as follows: (i) knowledge distillation [21, 41, 35, 39]; (ii) quantization [3, 54, 37, 12, 53, 55, 2, 10]; (iii) new architecture design [40, 19, 23, 22, 45]; and (iv) pruning.”
SqueezeLLM: Dense-and-Sparse Quantization (2023, methodology): “Quantization-Aware Training (QAT) requires retraining the model to adapt its weights to help recover accuracy after quantization [2, 35, 55, 82, 85, 86], whereas Post-Training Quantization (PTQ) performs quantization without any retraining [5, 44, 49, 56, 87].”
Post-Training Quantization for Vision Transformer (2021, methodology): “[35] propose TernaryBERT, which use both approximation-based and lossaware ternarization methods and empirically investigate the ternarization granularity of different parts of BERT.”