Abstractive Sentence Summarization with Attentive Recurrent Neural Networks

By Sumit Chopra, Michael Auli, Alexander M. Rush. 2016

933 citations recorded by the source corpus.

Abstractive Sentence Summarization generates a shorter version of a given sentence while attempting to preserve its meaning. We introduce a conditional recurrent neural network (RNN) which generates a summary of an input sentence. The conditioning is provided by a novel convolutional attention-based encoder which ensures that the decoder focuses on the appropriate input words at each step of generation. Our model relies only on learned features and is easy to train in an end-to-end fashion on large data sets. Our experiments show that the model signiﬁcantly outperforms the recently proposed state-of-the-art method on the Giga-word corpus while performing competitively on the DUC-2004 shared task.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Generating Wikipedia by Summarizing Long Sequences (2018, methodology): “RNN-based encoder-decoder models with attention (seq2seq) perform very well on this task in both ROUGE (Lin, 2004), an automatic metric often used in summarization, and human evaluation (Chopra et al., 2016). In Nallapati et al. (2016), an abstractive summarization dataset is proposed by modifying a”
Get To The Point: Summarization with Pointer-Generator Networks (2017, background): “However, the recent success of sequence-to-sequence models (Sutskever et al., 2014), in which recurrent neural networks (RNNs) both read and freely generate text, has made abstractive summarization viable (Chopra et al., 2016; Nallapati et al., 2016; Rush et al., 2015; Zeng et al., 2016).”
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond (2016, methodology): “html is compared with ABS and ABS+ models, RASElman from (Chopra et al., 2016), as well as TOPIARY, the top performing system on DUC-2004 in Table 2.”
WikiLingua: A New Benchmark Dataset for Multilingual Abstractive Summarization (2020, methodology): “Chopra et al. (2016) shows improvements over this model using a recurrent neural network for the decoder.”
MLSUM: The Multilingual Summarization Corpus (2020, methodology): “Gigaword Again using newswire as source data, the english Gigaword (Napoles et al., 2012; Rush et al., 2015; Chopra et al., 2016) corpus is characterized by its large size and the high diversity in terms of sources.”
Learning to summarize from human feedback (2020, background): “To make short-term progress towards this goal, we focus on abstractive English text summarization, as it has a long history in the NLP community [16, 8, 54, 59, 50], and is a subjective task where we believe it is difficult to quantify summary quality without human judgments.”
TL;DR: Mining Reddit to Learn Automatic Summarization (2017, background): “But with recent advancements in the field of deep learning, new ground was broken using various kinds of neural network models (Rush et al., 2015; Hu et al., 2015; Chopra et al., 2016; See et al., 2017).”
SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization (2019, background): “This requires natural language generation techniques with high level of semantic understanding (Chopra et al., 2016; Rush et al., 2015; Khandelwal et al., 2019; Zhang et al., 2019; See et al., 2017; Chen and Bansal, 2018; Gehrmann et al., 2018).”
OpenNMT: Open-Source Toolkit for Neural Machine Translation (2017, methodology): “Using OpenNMT, we were able to replicate the sentence summarization results of Chopra et al. (2016), reaching a ROUGE-1 score of 33.”
BLEURT: Learning Robust Metrics for Text Generation (2020, background): “, 2015) which can tackle a wide array of tasks including translation (Koehn, 2009), summarization (Mani, 1999; Chopra et al., 2016), structureddata-to-text generation (McKeown, 1992; Kukich, 1983; Wiseman et al.”
Learning to Write with Cooperative Discriminators (2018, background): “Language models based on Recurrent Neural Networks (RNNs) have brought substantial advancements across a wide range of language tasks (Jozefowicz et al., 2016; Bahdanau et al., 2015; Chopra et al., 2016).”
Bottom-Up Abstractive Summarization (2018, background): “Chopra et al. (2016) show that attention approaches that are more speciﬁc to summarization can further improve the performance of models.”
Controllable Abstractive Summarization (2017, methodology): “Research in abstractive summarization with sequence-to-sequence models focuses on neural architectures (Rush et al., 2015; Chopra et al., 2016; Nallapati et al., 2016; See et al., 2017; Paulus et al., 2017) and learning objectives (Paulus et al., 2017).”
Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies (2018, methodology): “Abstractive: Seq2Seq / Attention Sequence-to-sequence models with attention (Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2014) have been applied to various language tasks, including summarization (Chopra et al., 2016; Nallapati et al., 2016a).”
A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents (2018, methodology): “Neural abstractive summarization models have been studied in the past (Rush et al., 2015; Chopra et al., 2016; Nallapati et al., 2016) and later extended by source copying (Miao and Blunsom, 2016; See et al.”
Multi-Reward Reinforced Summarization with Saliency and Entailment (2018, methodology): “…made the ab-stractive summarization task more feasible and accurate, with recent ideas ranging from copy-pointer mechanism and redundancy coverage, to metric reward based reinforcement learning (Rush et al., 2015; Chopra et al., 2016; Ranzato et al., 2015; Nallapati et al., 2016; See et al., 2017).”
Neural Text Summarization: A Critical Evaluation (2019, methodology): “Gi-gaword (Graff and Cieri, 2003) is a set of articles and corresponding titles that was originally used for headline generation (Takase et al., 2016), but it has also been adapted to single-sentence summarization (Rush et al., 2015; Chopra et al., 2016).”
Abstractive Summarization of Reddit Posts with Multi-level Memory Networks (2018, methodology): “One of the most dominant architectures is to employ RNN-based seq2seq models with attention mechanism such as (Rush et al., 2015; Chopra et al., 2016; Nallap-ati et al., 2016; Cohan et al., 2018; Hsu et al., 2018; Gehrmann et al., 2018).”