Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding

By Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao. 2019

192 citations recorded by the source corpus.

This paper explores the use of knowledge distillation to improve a Multi-Task Deep Neural Network (MT-DNN) (Liu et al., 2019) for learning text representations across multiple natural language understanding tasks. Although ensemble learning can improve model performance, serving an ensemble of large DNNs such as MT-DNN can be prohibitively expensive. Here we apply the knowledge distillation method (Hinton et al., 2015) in the multi-task learning setting. For each task, we train an ensemble of different MT-DNNs (teacher) that outperforms any single model, and then train a single MT-DNN (student) via multi-task learning to \emph{distill} knowledge from these ensemble teachers. We show that the distilled MT-DNN significantly outperforms the original MT-DNN on 7 out of 9 GLUE tasks, pushing the GLUE benchmark (single model) to 83.7\% (1.5\% absolute improvement\footnote{ Based on the GLUE leaderboard at this https URL as of April 1, 2019.}). The code and pre-trained models will be made publicly available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

Language Models are Few-Shot Learners (2020, methodology): “Additionally, techniques like model distillation [LHCG19a] can further bring down the cost of such models, letting us adopt a paradigm of training single, large-scale models, then creating more efﬁcient versions of them for use in appropriate contexts.”
RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “QNLI: Recent submissions on the GLUE leaderboard adopt a pairwise ranking formulation for the QNLI task, in which candidate answers are mined from the training set and compared to one another, and a single (question, candidate) pair is classiﬁed as positive (Liu et al., 2019b,a; Yang et al., 2019).”
Multi-Task Deep Neural Networks for Natural Language Understanding (2019, background): “At last, we also would like to verify whether MT-DNN is resilience against adversarial attacks (Glockner et al., 2018; Talman and Chatzikyriakidis, 2018; Liu et al., 2019).”
A Hybrid Neural Network Model for Commonsense Reasoning (2019, methodology): “Most machine learning models can hardly outperform the naive baseline of majority voting (scored at 65.1) , including BERT (Devlin et al., 2018a) and Dis-tilled MT-DNN (Liu et al., 2019a).”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, methodology): “A crucial difference distinguishing our work from the STILTs, Snorkel MeTaL, and MT-DNN KD methods in Table 2 is that we do not single-task ﬁne-tune our model.”
Learning by Distilling Context (2022, background): “There has been a large literature on distilling knowledge in a neural network (Hinton et al., 2015b; Adriana et al., 2015; Liu et al., 2019; Yang et al., 2020; Xie et al., 2020).”
Patient Knowledge Distillation for BERT Model Compression (2019, methodology): “More recently, Liu et al. (2019a) used knowledge from ensemble models to improve single model performance on NLU tasks.”
Knowledge Distillation: A Survey (2020, methodology): “Furthermore, the parameters of the teacher model (or the connections between layers) also contain another knowledge (Liu et al. 2019c).”
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks (2019, methodology): “At the same time, Liu et al. (2019) has shown that multi-task learning can further improve a BERT-style pre-trained model.”
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020, background): “[20] distilled knowledge from ensemble models to improve the performance of a single model on NLU tasks.”
Pre-trained models for natural language processing: A survey (2020, methodology): “[113] exploited KD to improve a pre-trained multi-task deep neural network.”
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization (2019, methodology): “Following Liu et al. (2019b), we start with the default training/dev/test set of SNLI and SciTail.”
Knowledge Distillation from Internal Representations (2019, methodology): “…BERT (Devlin et al. 2018), GPT (Radford et al. 2018), GPT-2 (Radford et al. 2019), XLM (Lample and Conneau 2019), XLNet (Yang et al. 2019), and RoBERTa (Liu et al. 2019c) lead tasks such as text classiﬁcation, sentiment analysis, semantic role labeling, question answering, among others.”
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models (2020, background): “Limitations of existing work: Recent works (Liu et al., 2019; Zhu et al., 2019; Tang et al., 2019; Turc et al., 2019) leverage soft logits from teachers as optimization targets for distilling students, with some notable exceptions from concurrent work.”
Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System (2019, background): “ this way, the knowledge of complex models can be transferred to simple models. Distilling complex models into simple models has been shown to improve many NLP tasks to achieve impressive performance [14, 15, 18, 20]. Polino et al. [24] proposed a quantized distillation method. In their work, they i”
MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models (2019, methodology): “Following BERT, XLNet (Yang et al., 2019) proposes a generalized autoregressive pretraining method and RoBERTa (Liu et al., 2019c) optimizes BERT pretraining approach.”
TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval (2020, background): “MT-DNN ensemble [17], [40] and MKDN [38] improved the student model performance via learning from multiple teachers.”
Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution (2020, methodology): “KD has been implemented in many machine learning tasks, for example image classification [28], object detection [12, 61], video labeling [70], natural language processing [56, 41, 53, 36, 57], and speech recognition [11, 55, 37].”