MASS: Masked Sequence to Sequence Pre-training for Language Generation

By Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, Tie-Yan Liu. 2019

1,025 citations recorded by the source corpus.

Pre-training and fine-tuning, e.g., BERT, have achieved great success in language understanding by transferring knowledge from rich-resource pre-training task to the low/zero-resource downstream tasks. Inspired by the success of BERT, we propose MAsked Sequence to Sequence pre-training (MASS) for the encoder-decoder based language generation tasks. MASS adopts the encoder-decoder framework to reconstruct a sentence fragment given the remaining part of the sentence: its encoder takes a sentence with randomly masked fragment (several consecutive tokens) as input, and its decoder tries to predict this masked fragment. In this way, MASS can jointly train the encoder and decoder to develop the capability of representation extraction and language modeling. By further fine-tuning on a variety of zero/low-resource language generation tasks, including neural machine translation, text summarization and conversational response generation (3 tasks and totally 8 datasets), MASS achieves significant improvements over the baselines without pre-training or with other pre-training methods. Specially, we achieve the state-of-the-art accuracy (37.5 in terms of BLEU score) on the unsupervised English-French translation, even beating the early attention-based supervised model.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “This has led to the development of a wide variety of pre-training objectives [Dai and Le, 2015; Ramachandran et al., 2016; Radford et al., 2018; Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019b; Wang et al., 2019a; Song et al., 2019; Dong et al., 2019; Joshi et al., 2019].”
RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “However, newer methods have improved performance by multi-task ﬁne tuning (Dong et al., 2019), incorporating entity embeddings (Sun et al., 2019), span prediction (Joshi et al., 2019), and multiple variants of autoregressive pretraining (Song et al., 2019; Chan et al., 2019; Yang et al., 2019).”
Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “However, the decoder of MASS only predicted tokens that was masked in the encoder, whereas mBART reconstructs the full target sequence which allows to apply not only ‘‘masking’’ but any possible noise functions.”
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2019, methodology): “The largest improvements have come from pre-training on both source and target languages (Song et al., 2019; Lample & Conneau, 2019), but this requires pretraining on all languages of interest.”
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “We employ two techniques for adding noise: Randomly masking tokens Similar to Devlin et al. (2019) and Song et al. (2019), we randomly mask the input sequence tokens at training before feeding it to the encoder.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…(Sennrich et al., 2015; Edunov et al., 2018), language model fusion (Gulcehre et al., 2015; Sriram et al., 2017), self-supervised pre-training (Dai and Le, 2015; Ramachandran et al., 2016; Zhang and Zong, 2016; Song et al., 2019) and unsupervised NMT (Lample et al., 2017; Artetxe et al., 2017).”
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020, methodology): “, 2019) and sequence to sequence pretraining (Song et al., 2019).”
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, background): “Transformer-based sequence models has led to a dramatic improvement in performance when finetuned for both natural language understanding and text generation tasks (Radford et al., 2018a; Devlin et al., 2019; Rothe et al., 2019; Yang et al., 2019; Joshi et al., 2019; Song et al., 2019; Dong et al., ”
Microsoft Research Asia’s Systems for WMT19 (2019, methodology): “More details about MASS can be found in Song et al. (2019).”
Learning to summarize from human feedback (2020, background): “Finally, there has been extensive research on modifying architectures [22, 59] and pre-training procedures [70, 36, 49, 60, 53, 14] for improving summarization performance.”
PALM: Pre-training an Autoencoding&autoregressive Language Model for Context-conditioned Generation (2020, methodology): “MASS (Song et al., 2019) and BART (Lewis et al., 2019) are the two pre-training approaches most similar to PALM.”
Mask-Predict: Parallel Decoding of Conditional Masked Language Models (2019, methodology): “Concurrently with our work, Song et al. (2019) extend the approach of Lample and Conneau (2019) by using separate encoder and decoder parameters (as in our model) and pre-training them jointly in an autoregressive version of masked language modeling, although with monolingual data.”
On Faithfulness and Factuality in Abstractive Summarization (2020, background): “More advanced pretrained text generators (Radford et al., 2018, 2019; Dong et al., 2019; Song et al., 2019; Khandelwal et al., 2019; Rothe et al., 2020) are even better at capturing world knowledge as they are informed by a vast amount of background text.”
Revisiting Self-Training for Neural Sequence Generation (2019, background): “Notably, noisy ST in the full setting approaches the performance of state-of-the-art systems which use much larger datasets for pretraining (Song et al., 2019).”
Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, background): “Since the access to labeled parallel data is very limited, especially for low resource languages, better pretraining techniques that utilizes unlabeled data is the key to unlock better machine translation performances [14, 34, 51].”