PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization

By Jingqing Zhang, Yao Zhao, Mohammad Saleh, Peter J. Liu. 2019

2,535 citations recorded by the source corpus.

Recent work pre-training Transformers with self-supervised objectives on large text corpora has shown great success when fine-tuned on downstream NLP tasks including text summarization. However, pre-training objectives tailored for abstractive text summarization have not been explored. Furthermore there is a lack of systematic evaluation across diverse domains. In this work, we propose pre-training large Transformer-based encoder-decoder models on massive text corpora with a new self-supervised objective. In PEGASUS, important sentences are removed/masked from an input document and are generated together as one output sequence from the remaining sentences, similar to an extractive summary. We evaluated our best PEGASUS model on 12 downstream summarization tasks spanning news, science, stories, instructions, emails, patents, and legislative bills. Experiments demonstrate it achieves state-of-the-art performance on all 12 downstream datasets measured by ROUGE scores. Our model also shows surprising performance on low-resource summarization, surpassing previous state-of-the-art results on 6 datasets with only 1000 examples. Finally we validated our results using human evaluation and show that our model summaries achieve human performance on multiple datasets.

Subfields: Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Natural Language Processing

Citations in the corpus

TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, methodology): “Paraphrases were auto-generated using a PEGASUS-based paraphrasing model (Zhang et al., 2019; Rajauria, 2020), then filtered and edited to ensure that the meaning of the question was preserved.”
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, methodology): “(2021), a PEGASUS model (Zhang et al., 2020) pre-trained specifically for summarization.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “However, recent work by Fabbri et al. (2020) demonstrates that this may not hold in summarization, where the automated metric BERTScore (Zhang et al., 2020b) does not improve upon the correlation of ROUGE.”
Big Bird: Transformers for Longer Sequences (2020, methodology): “As has been pointed out recently [76, 108], pretraining helps in generative tasks, we warm start from our general purpose MLM pretraining on base-sized models as well as utilizing state-of-the-art summarization speciﬁc pretraining from Pegasus [108] on large-sized models.”
Learning to summarize from human feedback (2020, background): “Finally, there has been extensive research on modifying architectures [22, 59] and pre-training procedures [70, 36, 49, 60, 53, 14] for improving summarization performance.”
Longformer: The Long-Document Transformer (2020, methodology): “Note that the BigBird summarization model supports sequence length of 4K tokens but starts from and continues pre-training Pegasus (Zhang et al., 2020), a model”
Few-Shot Text Generation with Natural Language Instructions (2020, methodology): “We assume this is due to the fact that Zhang et al. (2020) introduce some modiﬁcations to their training procedure that encourage the model to copy text.”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “Another innovation building on monoBERT is the work of Zheng et al. [2020], who extended the pre-BERT NPRF (Neural Pseudo Relevance Feedback) approach [Li et al.”
HTLM: Hyper-Text Pre-Training and Prompting of Language Models (2021, methodology): “For our baseline, we compare against PEGA-SUS (Zhang et al., 2019), the current state of the art for zero shot summarization.”
Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, methodology): “For summarization, we also compare our method to WikiTransfer (Fabbri et al., 2021) and Pega-sus (Zhang et al., 2020) – both of which use data augmentation techniques tailored towards summarization.”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, methodology): “This approach is inspired by MASS [67] and PEGASUS [81].”
Sparse is Enough in Scaling Transformers (2021, methodology): “Several recent papers studied this dataset and tasks and it has been shown [46, 45] that pretraining on C4 yields significant improvements on this task.”
Pre-Trained Models: Past, Present and Future (2021, methodology): “TernaryBERT (Zhang et al., 2020b) applies knowledge distillation in quantization, forcing low-bit models to imitate full-precision models.”