Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System

By Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, Daxin Jiang. 2019

107 citations recorded by the source corpus.

Deep pre-training and fine-tuning models (such as BERT and OpenAI GPT) have demonstrated excellent results in question answering areas. However, due to the sheer amount of model parameters, the inference speed of these models is very slow. How to apply these complex models to real business scenarios becomes a challenging but practical problem. Previous model compression methods usually suffer from information loss during the model compression procedure, leading to inferior models compared with the original one. To tackle this challenge, we propose a Two-stage Multi-teacher Knowledge Distillation (TMKD for short) method for web Question Answering system. We first develop a general Q&A distillation task for student model pre-training, and further fine-tune this pre-trained student model with multi-teacher knowledge distillation on downstream tasks (like Web Q&A task, MNLI, SNLI, RTE tasks from GLUE), which effectively reduces the overfitting bias in individual teacher models, and transfers more general knowledge to the student model. The experiment results show that our method can significantly outperform the baseline methods and even achieve comparable results with the original teacher models, along with substantial speedup of model inference.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

Learning by Distilling Context (2022, background): “There has been a large literature on distilling knowledge in a neural network (Hinton et al., 2015b; Adriana et al., 2015; Liu et al., 2019; Yang et al., 2020; Xie et al., 2020).”
Knowledge Distillation: A Survey (2020, methodology): “…of them belong to natural language understanding (NLU), and many of these KD methods for NLU are designed as the task-speciﬁc distillation (Tang et al. 2019; Turc et al. 2019; Mou et al. 2016) and multi-task distillation (Liu et al. 2019f; Yang et al. 2020c; Sanh et al. 2019; Clark et al. 2019).”
Knowledge Distillation for Optimization of Quantized Deep Neural Networks (2019, background): “Some previous works on knowledge distillation tried to utilize the average of soft labels produced from multiple models [28], [29].”
Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals (2021, unknown): “[35] designed a multi-teacher knowledge distillation paradigm in a Web Question Answering system.”
Reinforced Multi-Teacher Selection for Knowledge Distillation (2020, methodology): “Yang et al. (2020) propose a two-stage multi-teacher KD method, which ﬁrst pre-trains a student model for the Q&A distillation task and then ﬁne-tunes the pre-trained student model with multi-teacher distillation on downstream tasks.”