Distilling Task-Specific Knowledge from BERT into Simple Neural Networks

By Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, Jimmy J. Lin. 2019

459 citations recorded by the source corpus.

In the natural language processing literature, neural networks are becoming increasingly deeper and complex. The recent poster child of this trend is the deep language representation model, which includes BERT, ELMo, and GPT. These developments have led to the conviction that previous-generation, shallower neural networks for language understanding are obsolete. In this paper, however, we demonstrate that rudimentary, lightweight neural networks can still be made competitive without architecture changes, external training data, or additional input features. We propose to distill knowledge from BERT, a state-of-the-art language representation model, into a single-layer BiLSTM, as well as its siamese counterpart for sentence-pair tasks. Across multiple datasets in paraphrasing, natural language inference, and sentiment classification, we achieve comparable results with ELMo, while using roughly 100 times fewer parameters and 15 times less inference time.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

TinyBERT: Distilling BERT for Natural Language Understanding (2019, methodology): “, 2020), knowledge distillation (Tang et al., 2019; Sanh et al., 2019; Turc et al., 2019; Sun et al., 2020; Liu et al., 2020; Wang et al., 2020), pruning (Cui et al.”
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (2020, background): “eorts in the NLU literature for distilling [15, 33], compressing [40], and pruning [20] BERT can be instrumental in narrowing this gap, they generally achieve signicantly smaller speedups than our redesigned architecture for IR, due to their generic nature, and more aggressive optimizations oen c”
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, background): “Tang et al. (2019) distill ﬁne-tuned BERT into an extremely small bidirectional LSTM. Turc et al. (2019a) initialize the student with a small pre-trained LM during task-speciﬁc distillation.”
The Right Tool for the Job: Matching Model and Instance Complexities (2020, background): “The result is typically a student that is as accurate as the teacher, but smaller and faster (Kim and Rush, 2016; Jiao et al., 2019; Tang et al., 2019; Sanh et al., 2019).”
Efficient Transformers: A Survey (2020, methodology): “For example, DistilBERT (Sanh et al., 2019), task-speciﬁc distillation (Tang et al., 2019) and TinyBERT (Jiao et al., 2019).”
A Primer in BERTology: What We Know About How BERT Works (2020, neutral): “, 2019) ×6§ 98% ×27§ mBERT3 CoNLL-2018 POS and morphology BiLSTM soft (Tang et al., 2019) ×110 91% ×434‡ BiLSTM1 MNLI, QQP, SST-2”
Knowledge Distillation: A Survey (2020, methodology): “…methods are proposed for solving the numerous NLP tasks (Liu et al. 2019b; Gordon and Duh 2019; Haidar and Rezagholizadeh 2019; Yang et al. 2020c; Tang et al. 2019; Hu et al. 2018; Sun et al. 2019; Nakashole and Flauger 2017; Jiao et al. 2020; Wang et al. 2018d; Zhou et al. 2019a; Sanh et al.…”
Well-Read Students Learn Better: The Impact of Student Initialization on Knowledge Distillation (2019, methodology): “For instance, Hu et al. (2018) uses ELMo embeddings, and (Chia et al., 2018; Tang et al., 2019) use context-independent word embeddings.”
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices (2020, background): “There have been some efforts that taskspecifically distill BERT into compact models (Turc et al., 2019; Tang et al., 2019; Sun et al., 2019; Tsai et al., 2019).”
Blockwise Self-Attention for Long Document Understanding (2019, methodology): “There are two recent efforts that apply knowledge distillation to BERT pre-training for reducing model size: TinyBERT (Jiao et al., 2019) distills BERT using a smaller Transformer, and Tang et al. (2019) distills BERT with a BiLSTM (Hochreiter and Schmidhuber, 1997).”
Pre-training Tasks for Embedding-based Large-scale Retrieval (2020, methodology): “For example, Tang et al. (2019) demonstrate initial success in distilling the BERT model into a two-tower model with BiLSTM as encoders.”
Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning (2020, methodology): “Tang et al. (2019) used BERT as a knowledge distillation teacher to compress relevant information into smaller Bi-LSTMs, while Kuncoro et al. (2019) took a similar distillation approach.”
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020, methodology): “[36] used a BiLSTM architecture to extract task-specific knowledge from BERT.”
Pre-trained models for natural language processing: A survey (2020, background): “However, not only decreasing parameters but also simplifying model structures from Transformer to RNN [177] or CNN [20] can reduce the computational complexity.”