Model Compression with Multi-Task Knowledge Distillation for Web-scale Question Answering System

By Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, Daxin Jiang. 2019

20 citations recorded by the source corpus.

Deep pre-training and fine-tuning models (like BERT, OpenAI GPT) have demonstrated excellent results in question answering areas. However, due to the sheer amount of model parameters, the inference speed of these models is very slow. How to apply these complex models to real business scenarios becomes a challenging but practical problem. Previous works often leverage model compression approaches to resolve this problem. However, these methods usually induce information loss during the model compression procedure, leading to incomparable results between compressed model and the original model. To tackle this challenge, we propose a Multi-task Knowledge Distillation Model (MKDM for short) for web-scale Question Answering system, by distilling knowledge from multiple teacher models to a light-weight student model. In this way, more generalized knowledge can be transferred. The experiment results show that our method can significantly outperform the baseline methods and even achieve comparable results with the original teacher models, along with significant speedup of model inference.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Well-Read Students Learn Better: The Impact of Student Initialization on Knowledge Distillation (2019, background): “Recent improvements in language understanding have been obtained through language model pre-training of expensive deep models on unlabeled text (Devlin et al., 2018; Yang et al., 2019b; Sun et al., 2019; Liu et al., 2019).”
Pruning a BERT-based Question Answering Model (2019, background): “While distillation (student-teacher) of BERT has produced notably smaller models, (Tang et al., 2019; Turc et al., 2019; Tsai et al., 2019; Yang et al., 2019a), the focus has been on sentence level annotation tasks that do not require long-range DistilBERT (Sanh et al., 2019) reports modest speedup…”