Patient Knowledge Distillation for BERT Model Compression

By Siqi Sun, Yu Cheng, Zhe Gan, Jingjing Liu. 2019

984 citations recorded by the source corpus.

Pre-trained language models such as BERT have proven to be highly effective for natural language processing (NLP) tasks. However, the high demand for computing resources in training such models hinders their application in practice. In order to alleviate this resource hunger in large-scale model training, we propose a Patient Knowledge Distillation approach to compress an original large model (teacher) into an equally-effective lightweight shallow network (student). Different from previous knowledge distillation methods, which only use the output from the last layer of the teacher network for distillation, our student model patiently learns from multiple intermediate layers of the teacher model for incremental knowledge extraction, following two strategies: (i) PKD-Last: learning from the last k layers; and (ii) PKD-Skip: learning from every k layers. These two patient distillation schemes enable the exploitation of rich information in the teacher’s hidden layers, and encourage the student model to patiently learn from and imitate the teacher through a multi-layer distillation process. Empirically, this translates into improved results on multiple NLP tasks with a significant gain in training efficiency, without sacrificing model accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “, 2019) or cause a significant degradation in accuracy (Liu et al., 2019a; Sun et al., 2019).”
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, background): “It has become common practice to pre-train large models and distill them down to smaller ones (Sun et al., 2019; Turc et al., 2019) for real applications.”
TinyBERT: Distilling BERT for Natural Language Understanding (2019, methodology): “Then following the verified implementation procedure, we trained a 4-layer BERT-PKD and a 4-layer DistilBERT as the baselines.”
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, methodology): “Following previous language model pre-training (Devlin et al., 2018; Liu et al., 2019) and task-agnostic pre-trained language model distillation (Sanh et al., 2019; Jiao et al., 2019; Sun et al., 2019b), we evaluate our distilled models on the extractive question answering and GLUE benchmark.”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Model compression has been explored from different aspects [25, 37, 39, 34, 43, 20, 24, 50, 18, 74, 40, 26, 55, 59, 28, 60, 68, 33, 14, 38, 31].”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “To remedy this, CBT (Sun et al., 2019a) uses a contrastive loss but still mainly for video representa- ∗ Equal contribution. tion learning alone, with text input only considered as side information.”
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT (2019, methodology): “There have also been attempts to use distillation to reduce large pre-trained Transformer models such as BERT [7] in [27, 28].”
Pre-Trained Models: Past, Present and Future (2021, methodology): “ERNIE-M (Ouyang et al., 2020) proposes back-translation masked language modeling (BTMLM), and expands the scale of parallel corpora through back-translation mechanisms.”
Knowledge Distillation: A Survey (2020, methodology): “To address this problem, several lightweight variations of BERT (called BERT model compression) using knowledge distillation are proposed (Sun et al., 2019; Jiao et al., 2019; Tang et al., 2019; Sanh et al., 2019).”
Training with Quantization Noise for Extreme Model Compression (2020, methodology): “Finally, knowledge distillation [6] has been applied to sentence representation [13, 27, 28, 29, 30], to reduce the size of a BERT model [31].”
FastBERT: a Self-distilling BERT with Adaptive Inference Time (2020, methodology): “PKD-BERT (Sun et al., 2019a) adopts an incremental extraction process that learns generalizations from intermediate layers of the teacher model.”
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices (2020, background): “Concurrently to our work, Sun et al. (2019) distill BERT into shallower students through knowledge distillation and an additional knowledge transfer of hidden states on multiple intermediate layers.”
Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed (2021, background): “Knowledge distillation (Bucilua et al., 2006; Hinton et al., 2015) has been explored extensively in supervised learning tasks such as image classiﬁcation [26, 46, 44, 7, 36, 43], language modeling [35, 37, 28, 14, 34], and speech recognition [3, 6, 40, 17, 9].”
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020, methodology): “However, the performance greatly relies on the design of the loss function [14, 33, 15].”
Pre-trained models for natural language processing: A survey (2020, background): “[169] introduced a “patient” teacherstudent mechanism, Liu et al.”
TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020, methodology): “…(Ma et al., 2019; Lan et al., 2020), weight-sharing (Dehghani et al., 2019; Lan et al., 2020), knowledge distillation (Sanh et al., 2019; Sun et al., 2019; Jiao et al., 2019 pruning (Michel et al., 2019; Voita et al., 2019; Fan et al., 2019), adaptive depth and/or width (Liu et al.,…”