ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation

By Dongling Xiao, Yukun Li, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang. 2020

135 citations recorded by the source corpus.

Current pre-training works in natural language generation pay little attention to the problem of exposure bias on downstream tasks. To address this issue, we propose an enhanced multi-flow sequence to sequence pre-training and fine-tuning framework named ERNIE-GEN, which bridges the discrepancy between training and inference with an infilling generation mechanism and a noise-aware generation method. To make generation closer to human writing patterns, this framework introduces a span-by-span generation flow that trains the model to predict semantically-complete spans consecutively rather than predicting word by word. Unlike existing pre-training methods, ERNIE-GEN incorporates multi-granularity target sampling to construct pre-training data, which enhances the correlation between encoder and decoder. Experimental results demonstrate that ERNIE-GEN achieves state-of-the-art results with a much smaller amount of pre-training data and parameters on a range of language generation tasks, including abstractive summarization (Gigaword and CNN/DailyMail), question generation (SQuAD), dialogue generation (Persona-Chat) and generative question answering (CoQA). The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE/ernie-gen.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Itsumi Saito
2. Yu Sun
3. Weizhen Qi
4. Weizhu Chen
5. Nan Duan
6. Dongling Xiao
7. Kosuke Nishida
8. Kyosuke Nishida

Citations in the corpus

• UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning (2020, methodology): “For the Seq2Seq generation task, we iteratively sample fragments from the token sequence until the 25% budget has been spent, inspired by Xiao et al. (2020).”
• GLGE: A New General Language Generation Evaluation Benchmark (2020, methodology): “…tasks, an increasing number of pretrained language models designed for Natural Language Generation (NLG) tasks have recently been proposed, such as MASS (Song et al., 2019), BERT-share (Rothe et al., 2020), BART (Lewis et al., 2020), ProphetNet (Qi et al., 2020), and ERINE-GEN (Xiao et al., 2020).”
• ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding (2020, methodology): “We employ T-test to extract semantically-complete n -grams statistically from unlabeled text corpora X (Xiao et al., 2020), as described in Algorithm 1.”
• Abstractive Summarization with Combination of Pre-trained Sequence-to-Sequence and Saliency Models (2020, methodology): “Dong et al. (2019) and Xiao et al. (2020) extended the BERT structure to handle seq-to-seq tasks.”
• Learning to Generate Questions by Learning to Recover Answer-containing Sentences (2021, unknown): “Open-GPT (Radford et al., 2018), BERT (Devlin et al., 2019), XLNet (Yang et al., 2019), PEGASUS (Zhang et al., 2019), ERNIE-GEN (Xiao et al., 2020), UniLM (Dong et al., 2019), UniLMv2 (Bao et al., 2020), T5 (Raffel et al., 2020) and BART (Lewis et al., 2020) utilize Transformer (Vaswani et al., 2017”