TinyBERT: Distilling BERT for Natural Language Understanding

By Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fangshan Wang, Qun Liu. 2019

2,430 citations recorded by the source corpus.

Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usually computationally expensive, so it is difficult to efficiently execute them on resource-restricted devices. To accelerate inference and reduce model size while maintaining accuracy, we first propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models. By leveraging this new KD method, the plenty of knowledge encoded in a large “teacher” BERT can be effectively transferred to a small “student” TinyBERT. Then, we introduce a new two-stage learning framework for TinyBERT, which performs Transformer distillation at both the pre-training and task-specific learning stages. This framework ensures that TinyBERT can capture the general-domain as well as the task-specific knowledge in BERT. TinyBERT4 with 4 layers is empirically effective and achieves more than 96.8% the performance of its teacher BERT-Base on GLUE benchmark, while being 7.5x smaller and 9.4x faster on inference. TinyBERT4 is also significantly better than 4-layer state-of-the-art baselines on BERT distillation, with only ~28% parameters and ~31% inference time of them. Moreover, TinyBERT6 with 6 layers performs on-par with its teacher BERT-Base.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “Some current work along these lines include distillation [Hinton et al., 2015; Sanh et al., 2019; Jiao et al., 2019], parameter sharing [Lan et al.”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, methodology): “These models were distilled with loss terms adapted from Sanh et al. (2019), Jiao et al. (2020), and Shleifer and Rush (2020), in order to match the predicted next-token distributions and latent states between these small models and BlenderBot3B during training.”
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (2020, background): “While ongoing efforts in the NLU literature for distilling [14, 33], compressing [40], and pruning [19] BERT can be instrumental in narrowing this gap, Query Document Query Encoder, fQ Document Encoder, fD MaxSim MaxSim MaxSim score”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “ TinyBERT proposed a distillation objective that additionally considers the mean squared error between the two models’ embedding layers, transformer hidden states, and transformer attention matrices [Jiao et al., 2019]. Gao et al. [2020b] observed that distillation can be applied to both a BERT mode”
WARP: Word-level Adversarial ReProgramming (2021, methodology): “Besides our best WARP models, we also include the human baselines, current state-of-the-art model (He et al., 2020), the regular ﬁne-tuned pretrained model we use, and also include relatively small language models, including (Jiao et al., 2020), (Clark et al., 2020), (Houlsby et al., 2019).”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “Related work may include distillation (Sanh et al., 2019; Jiao et al., 2020; Wang et al., 2020), quantization (Zafrir et al., 2019; Shen et al., 2020; Tao et al., 2022), and pruning (Michel et al., 2019; Fan et al., 2019).”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Model compression has been explored from diﬀerent aspects [25, 37, 39, 34, 43, 20, 24, 50, 18, 74, 40, 26, 55, 59, 28, 60, 68, 33, 14, 38, 31].”
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (2022, methodology): “We use the COCO image caption333 dataset (Lin et al., 2014) with the Karpathy334 split (Karpathy and Fei-Fei, 2017).”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, background): “Satellited around BERT, there are also studies on model compression (Sun et al., 2019c; Jiao et al., 2019; Shen et al., 2019) and extension from understanding to generation (Chen et al., 2019a; Clinchant et al., 2019; Wang and Cho, 2019).”
Accelerating Large Language Model Decoding with Speculative Sampling (2023, background): “…is heavily coupled with the model size in memory, quantisation to int8 or even int4 (Dettmers et al., 2022; Yao et al., 2022) and distillation (Jiao et al., 2020; Sanh et al., 2019) of transformers are eﬀective techniques for reducing sampling latency with little to no performance penalty.”
The Right Tool for the Job: Matching Model and Instance Complexities (2020, methodology): “Our experiments with a distilled version of BERT (Jiao et al., 2019) show that our method further improves the speed/accuracy curve on top of that model.”
Efficient Transformers: A Survey (2020, methodology): “For example, DistilBERT (Sanh et al., 2019), task-speciﬁc distillation (Tang et al., 2019) and TinyBERT (Jiao et al., 2019).”
A Primer in BERTology: What We Know About How BERT Works (2020, methodology): “This is achieved through experiments with loss functions (Sanh et al., 2019; Jiao et al., 2019), mimicking the activation patterns of individual portions of the teacher network (Sun et al., 2019a), and knowledge transfer at different stages at the pre-training (Turc et al., 2019; Jiao et al., 2019;…”
Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets (2020, methodology): “Our dense nearest neighbour model consists of a single BERT-base checkpoint and out-performs a BART-large model, and could be compressed using quantization and distillation techniques (Sanh et al., 2020; Jiao et al., 2019).”