BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
By Michael Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdel-rahman Mohamed, Omer Levy, Veselin Stoyanov, Luke S. Zettlemoyer. 2019
13,006 citations recorded by the source corpus.
We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and other recent pretraining schemes. We evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of sentences and using a novel in-filling scheme, where spans of text are replaced with a single mask token. BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It matches the performance of RoBERTa on GLUE and SQuAD, and achieves new state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains of up to 3.5 ROUGE. BART also provides a 1.1 BLEU increase over a back-translation system for machine translation, with only target language pretraining. We also replicate other pretraining schemes within the BART framework, to understand their effect on end-task performance.
Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Yejin Choi
- Graham Neubig
- Noah A. Smith
- Caiming Xiong
In Large Language Models
- Luke S. Zettlemoyer
- Hinrich Schütze
- Yejin Choi
- Timo Schick
- Mohit Bansal
- Furu Wei
- Jianfeng Gao
- Hannaneh Hajishirzi
In Natural Language Processing
- Mohit Bansal
- Luke S. Zettlemoyer
- Yejin Choi
- Graham Neubig
- Jianfeng Gao
- Percy Liang
- Hannaneh Hajishirzi
- Noah A. Smith
Citations in the corpus
- PaLM: Scaling Language Modeling with Pathways (2022, methodology): “, 2021b), or BART (Lewis et al., 2020), which are encoder-decoder models pre-trained with an infilling objective (masked LM or span corruption).”
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “We use BART-large [28], a pre-trained seq2seq transformer [52] with 400M parameters.”
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, methodology): “QA is based on a sequence-to-sequence network, pretrained on unsupervised data, such as T5 or BART (Raffel et al., 2019; Lewis et al., 2019).”
- REALM: Retrieval-Augmented Language Model Pre-Training (2020, background): “Alternatively, some recent work has proposed generation-based systems that apply a sequence-to-sequence model on x to directly generate y token-by-token (Lewis et al., 2019; Raffel et al., 2019).”
- UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “Our work is enabled by recent progress in textto-text neural architectures (Raffel et al., 2019; Lewis et al., 2019; Radford et al., 2019).”
- TTTTTackling WinoGrande Schemas (2020, positive): “Other than encoder-only pretrained transformer architectures [1, 3, 9], encoder–decoder style pretrained transformers [2, 5] have been proven to be effective in text generation tasks as well as comprehension tasks.”
- Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “mBART is trained by applying the BART (Lewis et al., 2019) to large-scale monolingual corpora across many languages.”
- Efficient Large Scale Language Modeling with Mixtures of Experts (2021, background): “Models are pretrained by hiding parts of the input: predicting the next word sequentially left-to-right, masking words in the text (Devlin et al., 2019; Liu et al., 2019), or perturbing and/or masking spans (Lewis et al., 2020; Raffel et al., 2020).”
- Reasoning Like Program Executors (2022, methodology): “EQUATE Table 10 presents performance break-down by subsets of EQUATE (Ravichander et al., 2019), where we compare P O E T -SQL BART and P O E T -SQL RoBERTa with their vanilla versions and previous baselines.”
- Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “Finetuning pretrained language models is a well-established method in NLP, with much of the work so far occurring on models in the range of 100M to 10B parameters (Dai & Le, 2015; Devlin et al., 2019; Raffel et al., 2020; Lewis et al., 2020, inter alia).”
- Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, background): “Denoising autoencoding (Vincent et al., 2008) trains a model that recovers original data from randomly corrupted versions, and is widely used as an effective self-supervised representation learning and pretraining strategy, e.g., in computer vision (Erhan et al., 2010) and natural language processin”
- The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “For example, XTREME (Hu et al., 2020) introduces a benchmark covering 40 languages across multiple NLU and retrieval tasks, XCOPA (Ponti et al., 2020) is a commonsense reasoning dataset for eleven languages, and MLQA (Lewis et al., 2020b) is a dataset for extractive question answering across seven l”
- Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (2021, methodology): “Predicted decompositions Analysis shows that BARTDECOMP’s decompositions are grammati- cal and well-structured.”
- mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “Popular models of this type are mBERT (Devlin, 2018), mBART (Liu et al., 2020), and XLM-R (Conneau et al., 2019), which are multilingual variants of BERT (Devlin et al., 2018), BART (Lewis et al., 2019a), and RoBERTa (Liu et al., 2019), respectively.”
- WikiLingua: A New Benchmark Dataset for Multilingual Abstractive Summarization (2020, methodology): “We use large, pre-trained language models as a starting point for our experiments, given their success on a variety of downstream Natural Language Processing tasks (Devlin et al., 2019), including state of the art results for text summarization (Lewis et al., 2019b; Liu and Lapata, 2019).”
- Big Bird: Transformers for Longer Sequences (2020, neutral): “[57] M.”