Unsupervised Pretraining for Sequence to Sequence Learning

By Prajit Ramachandran, Peter J. Liu, Quoc V. Le. 2016

292 citations recorded by the source corpus.

This work presents a general unsupervised learning method to improve the accuracy of sequence to sequence (seq2seq) models. In our method, the weights of the encoder and decoder of a seq2seq model are initialized with the pretrained weights of two language models and then fine-tuned with labeled data. We apply this method to challenging benchmarks in machine translation and abstractive summarization and find that it significantly improves the subsequent supervised models. Our main result is that pretraining improves the generalization of seq2seq models. We achieve state-of-the-art results on the WMT English→German task, surpassing a range of methods using both phrase-based machine translation and neural machine translation. Our method achieves a significant improvement of 1.3 BLEU from th previous best models on both WMT’14 and WMT’15 English→German. We also conduct human evaluations on abstractive summarization and find that our method outperforms a purely supervised learning baseline in a statistically significant manner.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “…noteworthy for NLP are algorithms for learning effective word embeddings (Collobert et al., 2011; Mikolov et al., 2013; Pen-nington et al., 2014) and language model pretraining (Dai and Le, 2015; Ramachandran et al., 2017; Peters et al., 2018; Howard and Ruder, 2018; Rad-ford et al., 2018).”
Dissecting Contextual Word Embeddings: Architecture and Representation (2018, background): “…resolution, and semantic role labeling (Peters et al., 2018), text classiﬁcation (Dai and Le, 2015; Howard and Ruder, 2018), sequence tagging (Peters et al., 2017), sequence–to–sequence learning (Ramachandran et al., 2017), and constituency parsing (Kitaev and Klein, 2018; Joshi et al., 2018).”
Learned in Translation: Contextualized Word Vectors (2017, background): “Ramachandran et al. [2016] propose initializing sequence-to-sequence models with pretrained language models and ﬁne-tuning for a speciﬁc task.”
Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, methodology): “Recently proposed techniques (Kiros et al., 2015; Ramachandran et al., 2016) ploiting discourse coherence information of this kind to train sentence encoders, but rely on generative objectives which require models to compute the likelihood of each word in a sentence at training time.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “We include the language modeling objective due to its historic use as a pre-training objective [Dai and Le, 2015; Ramachandran et al., 2016; Howard and Ruder, 2018; Radford et al., 2018; Peters et al., 2018] as well as its natural fit for the language model architectures we consider.”
Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “Qi et al. (2018) investigated the application of pre-trained word embeddings for MT; Ramachandran et al. (2017) proposed to pre-train the encoder-decoder modules as two separate language models.”
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “Decoder pre-training: Auto-regressive Language Modeling We pre-train the decoder with a standard auto-regressive language modeling (LM) objective similar to Ramachandran et al. (2017).”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, background): “Recent parallel work (Ramachandran et al., 2016; Howard & Ruder, 2018; Radford et al., 2018; Devlin et al., 2018; Liu et al., 2019b; Dai et al., 2019; Yang et al., 2019; Liu et al., 2019a; Lan et al., 2019) further builds upon these ideas by not just transferring the language model to extract…”
MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, positive): “ our proposed MASS. The token “ ” represents the mask symbol [M]. jpurkar et al., 2016). There are also some works pre-training the encoder-decoder model for language generation. Dai & Le (2015); Ramachandran et al. (2016) leverage a language model or auto-encoder to pre-train the encoder and de”
Cross-lingual Language Model Pretraining (2019, background): “Ramachandran et al. (2016) show that language modeling pretraining can also provide signiﬁcant improvements on machine translation tasks, even for high-resource language pairs such as English-German where there exists a signiﬁ-cant amount of parallel data.”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “, 2018), and weights of language models can be transferred to similar architectures (Ramachandran et al., 2017) and classification tasks (Howard and Ruder, 2018).”
A Simple Method for Commonsense Reasoning (2018, background): “hat Winograd Schema questions require more contextual information, hence our use of LMs instead of just word vectors. Neural LMs have also been applied successfully to improve downstream applications [18, 19, 20, 21]. In [18, 19, 20, 21], researchers have shown that pre-trained LMs can be used as fe”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…(Sennrich et al., 2015; Edunov et al., 2018), language model fusion (Gulcehre et al., 2015; Sriram et al., 2017), self-supervised pre-training (Dai and Le, 2015; Ramachandran et al., 2016; Zhang and Zong, 2016; Song et al., 2019) and unsupervised NMT (Lample et al., 2017; Artetxe et al., 2017).”
Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function (2019, methodology): “…weights from an auxiliary task is a widely used strategy that has shown to improve the performance in tasks such as text classiﬁcation (Dai and Le 2015; Howard and Ruder 2018), question-answering (Devlin et al. 2018), and machine translation (Ramachandran, Liu, and Le 2017; Qi et al. 2018).”
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, methodology): “Dai & Le (2015); Ramachandran et al. (2017) used LM and autoencoder pre-training on in-domain data to improve performance of RNN sequence models.”
Hierarchical Neural Story Generation (2018, methodology): “The two models can be leveraged together without architectural modiﬁcations: Ramachandran et al. (2016) use language models to initialize the encoder and decoder side of the seq2seq model independently, and Chorowski and Jaitly (2016) combine the predictions of the language model and seq2seq model…”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “, 2013), and has been ubiquitous in NLP (Mikolov et al., 2013; Pennington et al., 2014; Dai & Le, 2015; Ramachandran et al., 2016; Peters et al., 2018; Devlin et al., 2018; Brown et al., 2020).”
Unsupervised Neural Machine Translation (2017, methodology): “nnrich et al., 2016a). At the same time, Currey et al. (2017) showed that training an NMT system to directly copy target language text is also helpful and complementary with backtranslation. Finally, Ramachandran et al. (2017) pre-train the encoder and the decoder in language modeling. To the best o”
Masked Language Model Scoring (2019, neutral): “Beyond finetuning pretrained LMs and MLMs, monolingual pretraining has also improved NMT performance (Ramachandran et al., 2017; Conneau and Lample, 2019).”
When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation? (2018, background): “Researchers have worked on a number of methods for using monolingual data in NMT systems (Cheng et al., 2016; He et al., 2016; Ramachandran et al., 2016).”