On Extractive and Abstractive Neural Document Summarization with Transformer Language Models

By Jonathan Pilault, Raymond Li, Sandeep Subramanian, C. Pal. 2020

248 citations recorded by the source corpus.

We present a method to produce abstractive summaries of long documents that exceed several thousand words via neural abstractive summarization. We perform a simple extractive step before generating a summary, which is then used to condition the transformer language model on relevant information before being tasked with generating a summary. We also show that this approach produces more abstractive summaries compared to prior work that employs a copy mechanism while still achieving higher ROUGE scores. We provide extensive comparisons with strong baseline methods, prior state of the art work as well as multiple variants of our approach including those using only transformers, only extractive techniques and combinations of the two. We examine these models using four different summarization tasks and datasets: arXiv papers, PubMed papers, the Newsroom and BigPatent datasets. We find that transformer based methods produce summaries with fewer n-gram copies, leading to n-gram copying statistics that are more similar to human generated abstracts. We include a human evaluation, finding that transformers are ranked highly for coherence and fluency, but purely extractive methods score higher for informativeness and relevance. We hope that these architectures and experiments may serve as strong points of comparison for future work. Note: The abstract above was collaboratively written by the authors and one of the models presented in this paper based on an earlier draft of this paper.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Big Bird: Transformers for Longer Sequences (2020, neutral): “07 Extr-Abst-TLM [82] 41.”
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, background): “Table 1: Results of PEGASUSLARGE and PEGASUSBASE on all downstream datasets compared with the previous SOTA, which are fetched from (Lewis et al., 2019; Shi et al., 2019; Fabbri et al., 2019; Koupaee and Wang, 2018; Kim et al., 2019; Subramanian et al., 2019; Song et al., 2019; Zhang and Tetreault, ”
Recursively Summarizing Books with Human Feedback (2021, methodology): “…has been plenty of work on summarizing other kinds of long documents, such as scientiﬁc papers (Abu-Jbara and Radev, 2011; Collins et al., 2017; Subramanian et al., 2019; Cohan et al., 2018; Xiao and Carenini, 2019; Zhao et al., 2020; Sotudeh et al., 2020), and patents (Sharma et al., 2019),…”
A Divide-and-Conquer Approach to the Summarization of Academic Articles (2020, methodology): “Using a 3 years-old neural model [9] for summarizing the parts, our method achieves state-of-the-art results in two publicly available datasets of academic articles, surpassing recent more advanced models [14, 16].”
Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder (2023, methodology): “Summarization task (Rush et al., 2015; Chopra et al., 2016; Nallapati et al., 2016; Cheng and Lapata, 2016; See et al., 2017; Paulus et al., 2018; Pilault et al., 2020) can also be implemented with a Transformer architecture.”
DialogSum: A Real-Life Scenario Dialogue Summarization Dataset (2021, methodology): “, 2020), patents (Pilault et al., 2020) and academic papers (Koncel-Kedziorski et al.”
SEAL: Segment-wise Extractive-Abstractive Long-form Text Summarization (2020, methodology): “This differs from two-stage models [17, 18] where the extractive and abstractive stages use different encoders and trained separately.”
Generating SOAP Notes from Doctor-Patient Conversations Using Modular Summarization Techniques (2020, methodology): “In the space of two-step extractive-abstractive summarization approaches, Subramanian et al. (2019) summarize scientiﬁc papers by ﬁrst extracting sentences from it and then abstractively summarizing them.”
Can We Automate Scientific Reviewing? (2021, methodology): “Instead of regarding text generation as a holistic process, we decompose it into two steps, using an extract-then-generate paradigm (Chen and Bansal, 2018; Gehrmann et al., 2018; Subramanian et al., 2019; Dou et al., 2020).”
Efficient Attentions for Long Document Summarization (2021, methodology): “…include recent top-performing ab-stractive models: PEGASUS (Zhang et al., 2019), a large pre-trained summarization model with truncated inputs; TLM (Pilault et al., 2020), DANCER (Gidiotis and Tsoumakas, 2020), and SEAL (Zhao et al., 2020), all of them using hybrid extract-then-abstract methods;…”