A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents

By Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, Nazli Goharian. 2018

950 citations recorded by the source corpus.

Neural abstractive summarization models have led to promising results in summarizing relatively short documents. We propose the first model for abstractive summarization of single, longer-form documents (e.g., research papers). Our approach consists of a new hierarchical encoder that models the discourse structure of a document, and an attentive discourse-aware decoder to generate the summary. Empirical results on two large-scale datasets of scientific papers show that our model significantly outperforms state-of-the-art models.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, neutral): “PubMed, Arxiv (Kedzie et al., 2018; Cohan et al., 2018) 21.”
Big Bird: Transformers for Longer Sequences (2020, background): “This reduces its direct applicability to tasks that require larger context, like question answering [61], document summarization [21], etc.”
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, background): “The task is to generate the headline from the first sentence. arXiv, PubMed (Cohan et al., 2018) are two long document datasets of scientific publications from arXiv.org (113k) and PubMed (215k).”
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023, methodology): “Data We evaluate on summarization datasets CNN/Daily Mail (Nallapati et al., 2016), arXiv and PubMed (Cohan et al., 2018), MediaSum (Zhu et al., 2021), and Multi-News (Fabbri et al., 2019); (Google, 2020).”
SCROLLS: Standardized CompaRison Over Long Language Sequences (2022, methodology): “The reports and their summaries are longer than their equivalents in other popular long-document summarization datasets; for example, GovReport’s documents are approximately 1.5 and 2.5 times longer than the documents in arXiv and PubMed (Cohan et al., 2018), respectively.”
Longformer: The Long-Document Transformer (2020, methodology): “We evaluate LED on the summarization task using the arXiv summarization dataset (Cohan et al., 2018) which focuses on long document summarization in the scientific domain.”
Recursively Summarizing Books with Human Feedback (2021, methodology): “…on summarizing other kinds of long documents, such as scientiﬁc papers (Abu-Jbara and Radev, 2011; Collins et al., 2017; Subramanian et al., 2019; Cohan et al., 2018; Xiao and Carenini, 2019; Zhao et al., 2020; Sotudeh et al., 2020), and patents (Sharma et al., 2019), as well as multi-document…”
Sparse is Enough in Scaling Transformers (2021, methodology): “using the dataset of scientific papers from arXiv3[6].”
Bottom-Up Abstractive Summarization (2018, methodology): “For example, Cohan et al. (2018) use a hierarchical attention to detect”
Muppet: Massive Multi-task Representations with Pre-Finetuning (2021, methodology): “To do so, we ﬁnetune our models on a set of tasks including (1) ANLI (Nie et al., 2019) and Hyperpartisan (Kiesel et al., 2019) for classiﬁcation, (2) Arxiv (He et al., 2019), PubMed (Cohan et al., 2018), (Sharma et al., 2019) for structured prediction from the Penn Treebank dataset (Marcus et al., ”
Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model (2019, methodology): “…Lapata, 2016; Narayan et al., 2018b) and abstractive methods (Chopra et al., 2016; Nal-lapati et al., 2016b; See et al., 2017; Paulus et al., 2017; Cohan et al., 2018; C¸elikyilmaz et al., 2018; Gehrmann et al., 2018) In addition to the multi-document methods described above which address data…”
Neural Text Summarization: A Critical Evaluation (2019, methodology): “Red-dit TIFU (Kim et al., 2018) is a collection of posts scraped from Reddit where users post their daily stories and each post is required to contain a Too Long; Didn’t Read (TL;DR) summary.”
Abstractive Summarization of Reddit Posts with Multi-level Memory Networks (2018, methodology): “Moreover, the performance gaps between abstractive and extractive methods are much larger in our dataset than in other datasets (See et al., 2017; Paulus et al., 2018; Cohan et al., 2018), which means too that our dataset is highly abstractive.”
BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization (2019, background): “PUBMED (Cohan et al., 2018) contain scientific articles.”
On Extractive and Abstractive Neural Document Summarization with Transformer Language Models (2020, methodology): “A model using a pointing mechanism (we used our own implementation of the model developed by Cohan et al. (2018)) copies more than our transformer model, especially for higher n -grams.”
CTRLsum: Towards Generic Controllable Text Summarization (2020, methodology): “We use pretrained BART (Lewis et al., 2020) as the underlying architecture and perform experiments on three datasets: CNN/Dailymail news articles (Hermann et al., 2015), arXiv scientific papers (Cohan et al., 2018), and BIGPATENT patent documents (Sharma et al., 2019).”
Don’t Say What You Don’t Know: Improving the Consistency of Abstractive Summarization by Constraining Beam Search (2022, background): “abstracts as paper summaries (Cohan et al., 2018); headlines as news summaries (Narayan et al.”
QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization (2021, background): “…al., 2019; Jia et al., 2020) investigate how to generate better summaries on news article data, such as CNN/DailyMail (Hermann et al., 2015), Newsroom (Grusky et al., 2018), etc. Sci-entiﬁc paper summarization is another important branch (Cohan et al., 2018; Yasunaga et al., 2019; An et al., 2021).”
PRIMER: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization (2021, background): “ArXiv (Cohan et al., 2018) is a single document summarization dataset in the scientific paper domain.”