MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models
By Linqing Liu, Haiquan Wang, Jimmy J. Lin, Richard Socher, Caiming Xiong. 2019
24 citations recorded by the source corpus.
Pretrained language models have led to significant performance gains in many NLP tasks. However, the intensive computing resources to train such models remain an issue. Knowledge distillation alleviates this problem by learning a light-weight student model. So far the distillation approaches are all task-specific. In this paper, we explore knowledge distillation under the multi-task learning setting. The student is jointly distilled across different tasks. It acquires more general representation capacity through multi-tasking distillation and can be further fine-tuned to improve the model in the target domain. Unlike other BERT distillation methods which specifically designed for Transformer-based architectures, we provide a general learning framework. Our approach is model agnostic and can be easily applied on different future teacher model architectures. We evaluate our approach on a Transformer-based and LSTM based student model. Compared to a strong, similarly LSTM-based approach, we achieve better quality under the same computational constraints. Compared to the present state of the art, we reach comparable results with much faster inference speed.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Soham De
- Razvan Pascanu
- Samuel L. Smith
- Yoshua Bengio
- Caglar Gulcehre
- Quoc V. Le
- Kyunghyun Cho
- Mohit Bansal
Citations in the corpus
- Resurrecting Recurrent Neural Networks for Long Sequences (2023, background): “RNNs, which scale linearly with the sequence length, are therefore typically faster than transformers at inference time even for modest sequence lengths (Liu et al., 2019).”