Distilling Knowledge Learned in BERT for Text Generation

By Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingjing Liu. 2019

143 citations recorded by the source corpus.

Large-scale pre-trained language model such as BERT has achieved great success in language understanding tasks. However, it remains an open question how to utilize BERT for language generation. In this paper, we present a novel approach, Conditional Masked Language Modeling (C-MLM), to enable the finetuning of BERT on target generation tasks. The finetuned BERT (teacher) is exploited as extra supervision to improve conventional Seq2Seq models (student) for better text generation performance. By leveraging BERT’s idiosyncratic bidirectional nature, distilling knowledge learned in BERT can encourage auto-regressive Seq2Seq models to plan ahead, imposing global sequence-level supervision for coherent text generation. Experiments show that the proposed approach significantly outperforms strong Transformer baselines on multiple language generation tasks such as machine translation and text summarization. Our proposed model also achieves new state of the art on IWSLT German-English and English-Vietnamese MT datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Societal Biases in Language Generation: Progress and Challenges (2021, background): “, 2019) can also be used for auto-regressive generation (Wang and Cho, 2019; Chen et al., 2020), traditional auto-regressive models are still typically of better quality and more widely used for generation (Shwartz et al.”
Knowledge Distillation: A Survey (2020, background): “A variety of online knowledge distillation methods have been proposed, especially in the last few years (Zhang et al. 2018b; Chen et al. 2020a; Xie et al. 2019; Anil et al. 2018; Kim et al. 2019b; Zhou et al. 2018; Walawalkar et al. 2020; Wu and Gong 2021; Zhang et al. 2021a).”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “The recent progress focuses on parameter-efficient techniques for reducing trainable parameters, including adapter-based fine-tuning [25, 26, 27] that trains additional small residual blocks between layers, freezing-based fine-tuning [71, 18, 72] that freezes most of the pretrained parameters, and d”
Distilling the Knowledge of BERT for Sequence-to-Sequence ASR (2020, methodology): “Meanwhile, some studies have leveraged right context during the seq2seq model training by distilling the knowledge of a “bidirectional” teacher model [22, 23].”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “[26] leveraged BERT as the teacher model that generates sequences of word probability logits and treated the Seq2Seq model as the student network, which can effectively learn from the teacher’s outputs.”