Pre-training via Paraphrasing

By Michael Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida I. Wang, Luke S. Zettlemoyer. 2020

165 citations recorded by the source corpus.

We introduce MARGE, a pre-trained sequence-to-sequence model learned with an unsupervised multi-lingual multi-document paraphrasing objective. MARGE provides an alternative to the dominant masked language modeling paradigm, where we self-supervise the reconstruction of target text by retrieving a set of related texts (in many languages) and conditioning on them to maximize the likelihood of generating the original. We show it is possible to jointly learn to do retrieval and reconstruction, given only a random initialization. The objective noisily captures aspects of paraphrase, translation, multi-document summarization, and information retrieval, allowing for strong zero-shot performance on several tasks. For example, with no additional task-specific training we achieve BLEU scores of up to 35.8 for document translation. We further show that fine-tuning gives strong performance on a range of discriminative and generative tasks in many languages, making MARGE the most generally applicable pre-training method to date.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Distilling Knowledge from Reader to Retriever for Question Answering (2020, background): “Similarly, Lewis et al. (2020a) introduced a sequence-to-sequence model that is pre-trained by generating a target text, after retrieving a set of related texts.”
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, background): “, 2020b) Encoder-decoder 680M 25 Common Crawl (CC25) MARGE (Lewis et al., 2020a) Encoder-decoder 960M 26 Wikipedia or CC-News mT5 (ours) Encoder-decoder 300M – 13B 101 Common Crawl (mC4)”
The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, background): “KI-NLP tasks are usually solved with retriever-reader systems: ﬁrst, a retriever surfaces a small set of relevant documents from the knowledge source, then a reader uses the context to generate an answer (Chen et al., 2017; Lewis et al., 2020b; Guu et al., 2020).”
Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “Similarly, Lewis et al. (2020) proposed to jointly learn a retriever and a generative seq2seq model, using self-supervised training.”
KILT: a Benchmark for Knowledge Intensive Language Tasks (2020, methodology): “We experiment with ﬁne-tuning RAG (Lewis et al., 2020b) on KILT tasks, establishing a strong base-line on all of them.”
Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, methodology): “Following MARGE (Lewis et al., 2020a), we divide CC-News into shards based on document publishing dates, and restrict the space of candidate summaries for a given document to be from the same shard.”
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (2020, background): “Pre-trained language models (Radford et al., 2019; Devlin et al., 2018; Liu et al., 2019; Lewis et al., 2019; 2020) provide the defacto initialization for modeling most existing NLP tasks.”
Hurdles to Progress in Long-form Question Answering (2021, neutral): “Simple baselines such as just repeatedly copying the question, or choosing a random training set answer, can outperform LFQA systems such as RAG (Lewis et al., 2020c) in terms of ROUGE-L.”
Support-set bottlenecks for video-text representation learning (2020, background): “Subsequent works (Clark et al., 2020; Lewis et al., 2020a;b; Radford et al., 2019; Raﬀel et al., 2019), have improved upon the transformer architecture or training objective to learn even better representations.”
Better Fine-Tuning by Reducing Representational Collapse (2020, methodology): “Pre-trained langauge models (Radford et al., 2019; Devlin et al., 2018; Liu et al., 2019; Lewis et al., 2019; 2020) have been shown to capture a wide array of semantic, syntactic and world knowledge (Clark et al., 2019), and provide the defacto initialization for modeling most existing NLP tasks.”
End-to-End Training of Neural Retrievers for Open-Domain Question Answering (2021, methodology): “Furthermore, motivated by recent work (Guu et al., 2020; Lewis et al., 2020a), we also explore two approaches for end-to-end supervised training of the reader and retriever components.”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, background): “Augmenting traditional training with a non-parametric retrieval component has recently shown impressive results in pre-training (Khandelwal et al., 2019; Guu et al., 2020; Lewis et al., 2020a) and QA (Izacard and Grave, 2020; Karpukhin et al., 2020).”
UnitedQA: A Hybrid Approach for Open Domain Question Answering (2021, methodology): “In this work, we study two paradigms of reader modules, i.e. extractive (Karpukhin et al., 2020; Guu et al., 2020) and generative (Lewis et al., 2020; Izacard and Grave, 2021) readers.”
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining (2021, background): “The benefits of sentence level pretraining task are usually observed on some specific tasks (Chi et al., 2020; Lewis et al., 2020) such as modeling long-form texts (Ravula et al.”
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models (2021, unknown): “There are many multilingual question answering datasets such as XQuAD (Artetxe et al., 2020), TiDy (Clark et al., 2020), MKQA (Longpre et al., 2020) and MLQA (Lewis et al., 2020b).”
MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages (2022, background): “…(mBERT (Devlin, 2018), RoBERTa (Liu et al., 2019b), XLM (Lample and Conneau, 2019), XLM-R (Conneau et al., 2020), mBART (Liu et al., 2020), MARGE (Lewis et al., 2020), and mT5 (Xue et al., 2021) pre-trained on massively multilingual corpora have allowed for significant improvements in supporting…”
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models (2024, unknown): “In this part, we will briefly review studies with these two types of generators: parameter-accessible (white-box) and parameter-inaccessible (black-box).”
How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering (2020, background): “…has found that LMs’ factual predictions can be improved if more context is provided (Petroni et al., 2020a), which has inspired many retrieval-augmented LMs that retrieve evidence from external resources and condition the LMs’ prediction on this evidence (Guu et al., 2020; Lewis et al., 2020a,c).”