Pre-trained language model representations for language generation

By Sergey Edunov, Alexei Baevski, Michael Auli. 2019

140 citations recorded by the source corpus.

Pre-trained language model representations have been successful in a wide range of language understanding tasks. In this paper, we examine different strategies to integrate pre-trained representations into sequence to sequence models and apply it to neural machine translation and abstractive summarization. We find that pre-trained representations are most effective when added to the encoder network which slows inference by only 14%. Our experiments in machine translation show gains of up to 5.3 BLEU in a simulated resource-poor setup. While returns diminish with more labeled data, we still observe improvements when millions of sentence-pairs are available. Finally, on abstractive summarization we achieve a new state of the art on the full text version of CNN/DailyMail.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, neutral): “Existing approaches (Lample and Conneau, 2019; Edunov et al., 2019; Lewis et al., 2019; Raffel et al., 2019) have been proposed either to partially pre-train the model or to only focus on English corpora.”
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2019, background): “Previous work Edunov et al. (2019) has shown that models can be improved by incorporating pre-trained encoders, but gains from using pre-trained language models in de-coders have been limited.”
Unified Language Model Pre-training for Natural Language Understanding and Generation (2019, methodology): “We remove duplicated trigrams in beam search, and tweak the maximum summary length on the development set (Paulus et al., 2018; Edunov et al., 2019).”
PALM: Pre-training an Autoencoding&autoregressive Language Model for Context-conditioned Generation (2020, methodology): “Edunov et al. (Edunov et al., 2019) examine different strategies (e.g., ELMo) to add contex-tualized embeddings to sequence-to-sequence models, and observe the most improvement by adding the learned embeddings to the encoder.”
Improving Transformer-based Speech Recognition Using Unsupervised Pre-training (2019, methodology): “ognition systems when labeled data are limited. Recently, unsupervised pre-training has shown promising results in several areas, including Computer Vision (CV) [1], Natural Language Processing (NLP) [2, 3] and so on [4, 5, 6]. One work that stands out among these unsupervised pre-training methods i”
vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations (2019, background): “on.1 1 INTRODUCTION Self-supervised representation learning has led to large improvements in natural language processing (Peters et al.,2018;Radford et al.,2018;Baevski et al.,2019;Devlin et al.,2018;Edunov et al., 2019;Lample & Conneau,2019;Yang et al.,2019;Liu et al.,2019). The same topic has ”
Reducing Transformer Depth on Demand with Structured Dropout (2019, methodology): “Model Enc Dec ROUGE-1 ROUGE-2 ROUGE-L Abstractive Summarization Transformer (Edunov et al., 2019) 6 6 40.”
wav2vec: Unsupervised Pre-training for Speech Recognition (2019, neutral): “, 2019) improved many tasks such as text classification, phrase structure parsing and machine translation (Edunov et al., 2019; Lample & Conneau, 2019).”
Text Summarization with Pretrained Encoders (2019, background): “Very recently, there have been attempts to apply pretrained models to various generation problems (Edunov et al., 2019; Rothe et al., 2019).”
Masked Language Model Scoring (2019, methodology): “This handicapped their use in generative tasks, where they at best served to initialize encoderdecoder models (Edunov et al., 2019; Yang et al., 2019a) or unidirectional LMs (Wang et al.”
Pre-trained models for natural language processing: A survey (2020, methodology): “tialize the entire encoder and decoder by a multilingual pre-trained BERT model and shows a signiﬁcant improvement can be achieved on unsupervised MT and English-Romanian supervised MT. Edunov et al. [37] used ELMo to set the word embedding layer in the NMT model. This work shows performance improve”
Encoder-Agnostic Adaptation for Conditional Language Generation (2019, methodology): “More related, Edunov et al. (2019) experiment with a representation-based approach for applying ELMo (Peters et al., 2018) to the source and target sides of a standard seq2seq model separately.”
Better Document-Level Machine Translation with Bayes’ Rule (2019, methodology): “…is another area where pretrained language models have been applied, with existing work focusing on fine-tuning for repurposing an unconditional language model (Zhang et al., 2019; Edunov et al., 2019; Song et al., 2019; Dong et al., 2019; Ziegler et al., 2019; de Oliveira and Rodrigo, 2019).”
Pretrained Language Models for Dialogue Generation with Multiple Input Sources (2020, methodology): “Researchers have then utilized them in dialogue generation tasks (Budzianowski and Vulić, 2019; Edunov et al., 2019; Zhang et al., 2019).”
On Learning Universal Representations Across Languages (2020, methodology): “, 2019) or using pre-trained models to initialize encoders (Edunov et al., 2019; Zhang et al., 2019).”
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries (2021, background): “Pre-trained language models have achieved promising results in abstractive text summarization (Edunov et al., 2019; Dong et al., 2019; Song et al., 2019; Zhang et al., 2019, 2020).”
Putting Machine Translation in Context with the Noisy Channel Model (2019, methodology): “…area where pretrained language models have been applied to, with existing work focusing on ﬁne-tuning approaches for repurposing an unconditional language model (Zhang et al., 2019; Edunov et al., 2019; Song et al., 2019; Dong et al., 2019; Ziegler et al., 2019; de Oliveira & Rodrigo, 2019).”
Multimodal Conditionality for Natural Language Generation (2021, background): “Unconditional language models can be adopted for text generation tasks such as language translation (Edunov et al., 2019), question answering (Su et al., 2019a), and summarization (Zhang et al., 2019).”
The Summary Loop: Learning to Write Abstractive Summaries Without Examples (2020, methodology): “Performance can further be improved by constraining the attention (Gehrmann et al., 2018; Gui et al., 2019; Wang et al., 2019) and using pretrained Transformer-based language models (Lewis et al., 2019; Chi et al., 2019; Edunov et al., 2019).”