Unified Language Model Pre-training for Natural Language Understanding and Generation

By Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, M. Zhou, Hsiao-Wuen Hon. 2019

1,676 citations recorded by the source corpus.

This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks. The model is pre-trained using three types of language modeling tasks: unidirectional, bidirectional, and sequence-to-sequence prediction. The unified modeling is achieved by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction conditions on. UniLM compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks. Moreover, UniLM achieves new state-of-the-art results on five natural language generation datasets, including improving the CNN/DailyMail abstractive summarization ROUGE-L to 40.51 (2.04 absolute improvement), the Gigaword abstractive summarization ROUGE-L to 35.75 (0.86 absolute improvement), the CoQA generative question answering F1 score to 82.5 (37.1 absolute improvement), the SQuAD question generation BLEU-4 to 22.12 (3.75 absolute improvement), and the DSTC7 document-grounded dialog response generation NIST-4 to 2.67 (human performance is 2.65). The code and pre-trained models are available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “This has led to the development of a wide variety of pre-training objectives [Dai and Le, 2015; Ramachandran et al., 2016; Radford et al., 2018; Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019b; Wang et al., 2019a; Song et al., 2019; Dong et al., 2019; Joshi et al., 2019].”
RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “However, newer methods have improved performance by multi-task fine tuning (Dong et al., 2019), incorporating entity embeddings (Sun et al.”
Adversarial Training for Large Neural Language Models (2020, background): “, 2018) and other transformer-based models (Liu et al., 2019c; Radford et al., 2018; Clark et al., 2020; Dong et al., 2019; Bao et al., 2020).”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, methodology): “UNILM (Dong et al., 2019) uses multiple language modeling objectives – uni-directional (both left-to-right and right-to-left), bidirectional, and sequence-to-sequence prediction – to aid generation tasks like summarization and question generation.”
MLSUM: The Multilingual Summarization Corpus (2020, methodology): “METEOR is often reported in summarization papers (See et al., 2017; Dong et al., 2019) in addition to ROUGE.”
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, background): “However, the combination of pre-training with much larger external text corpora (such as Wikipedia, books, or Web-pages) and Transformerbased sequence models has led to a dramatic improvement in performance when fine-tuned for both natural language understanding and text generation tasks (Radford et”
Learning to summarize from human feedback (2020, background): “A human judges which is a better summary of the post.”
ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, methodology): “Results For both tasks, we used data and evaluation metrics released by Dong et al. [9].”
Discriminative Adversarial Search for Abstractive Summarization (2020, positive): “, 2018; Dong et al., 2019). However, external rules are usually added to further constrain the generation, like the filtering mechanism for copy described above (Gehrmann et al., 2018) or the inclusion of a length penalty factor (Wu et al., 2016). Hokamp & Liu (2017) reported improvements when addin”
PALM: Pre-training an Autoencoding&autoregressive Language Model for Context-conditioned Generation (2020, methodology): “62 UniLMLARGE (Dong et al., 2019) 43.”
Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering (2019, background): “Similar methods have also been applied in some very recent concurrent works (Dong et al., 2019; Alberti et al., 2019) on SQuADv2.”
How many data points is a prompt worth? (2021, neutral): “The main paradigm for adapting pretrained models for classification (Radford, 2018; Dong et al., 2019; Devlin et al., 2018) is fine-tuning via an explicit classifier head.”
BEiT: BERT Pre-Training of Image Transformers (2021, background): “Various MLM variants (Liu et al., 2019; Dong et al., 2019; Yang et al., 2019; Bao et al., 2020) are proposed to improve performance of Transformer pre-training.”
UL2: Unifying Language Learning Paradigms (2022, methodology): “In terms of individual tasks, UniLM and SCLM both outperforms T5 on 6 out of 9 tasks.”