Multi-stage Pretraining for Abstractive Summarization

By Sebastian Goodman, Zhenzhong Lan, Radu Soricut. 2019

6 citations recorded by the source corpus.

Neural models for abstractive summarization tend to achieve the best performance in the presence of highly specialized, summarization specific modeling add-ons such as pointer-generator, coverage-modeling, and inferencetime heuristics. We show here that pretraining can complement such modeling advancements to yield improved results in both short-form and long-form abstractive summarization using two key concepts: full-network initialization and multi-stage pretraining. Our method allows the model to transitively benefit from multiple pretraining tasks, from generic language tasks to a specialized summarization task to an even more specialized one such as bullet-based summarization. Using this approach, we demonstrate improvements of 1.05 ROUGE-L points on the Gigaword benchmark and 1.78 ROUGE-L points on the CNN/DailyMail benchmark, compared to a randomly-initialized baseline.

Subfields: Artificial Intelligence, Summarization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, neutral): “BIGPATENT, arXiv, PubMed and Multi-News datasets contain very long summaries and we truncate them to 256 tokens, in similar range compared to (Sharma et al., 2019; Cohan et al., 2018; Fabbri et al., 2019; Goodman et al., 2019).”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, methodology): “On one hand, this is due to advances in architectures and modeling that are mainly inspired by BERT and similar models in natural language understanding and generation [24, 50, 77, 43, 25, 62].”