Distilling the Knowledge of BERT for Text Generation

By Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingjing Liu. 2019

31 citations recorded by the source corpus.

Large-scale pre-trained language model, such as BERT, has recently achieved great success in a wide range of language understanding tasks. However, it remains an open question how to utilize BERT for text generation tasks. In this paper, we present a novel approach to addressing this challenge in a generic sequence-to-sequence (Seq2Seq) setting. We first propose a new task, Conditional Masked Language Modeling (C-MLM), to enable fine-tuning of BERT on target text-generation dataset. The fine-tuned BERT (i.e., teacher) is then exploited as extra supervision to improve conventional Seq2Seq models (i.e., student) for text generation. By leveraging BERT's idiosyncratic bidirectional nature, distilling the knowledge learned from BERT can encourage auto-regressive Seq2Seq models to plan ahead, imposing global sequence-level supervision for coherent text generation. Experiments show that the proposed approach significantly outperforms strong baselines of Transformer on multiple text generation tasks, including machine translation (MT) and text summarization. Our proposed model also achieves new state-of-the-art results on the IWSLT German-English and English-Vietnamese MT datasets.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

1. Zhe Gan
2. Jingjing Liu
3. Yu Cheng
4. Linjie Li
5. Yen-Chun Chen
6. J. Ainslie
7. Licheng Yu
8. Anirudh Ravula

Citations in the corpus

• Big Bird: Transformers for Longer Sequences (2020, background): “The model is easily adapted for a diverse range of sequence based tasks – as a seq2seq model for translation [92], summarization [66], generation [15], etc. or as a standalone encoders for sentiment analysis [84], POS tagging [65], machine reading comprehension [94], etc. – and it is known to…”
• Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “Satellited around BERT, there are also studies on model compression (Sun et al., 2019c; Jiao et al., 2019; Shen et al., 2019) and extension from understanding to generation (Chen et al., 2019a; Clinchant et al., 2019; Wang and Cho, 2019).”
• Adaptive Adapters: An Efficient Way to Incorporate BERT Into Neural Machine Translation (2021, methodology): “distillation [30], [31] is applied in [11]–[13] to transfer the knowledge from BERT to either the encoder [12] or decoder”
• Task-Oriented Feature Distillation (2020, methodology): “Recently, remarkable achievements have been attained with deep neural networks in all kinds of applications such as nature language processing [3, 13, 8, 60] and computer vision [54, 52, 51].”