A Neural Attention Model for Abstractive Sentence Summarization

By Alexander M. Rush, Sumit Chopra, Jason Aaron Edward Weston. 2015

2,850 citations recorded by the source corpus.

Summarization based on text extraction is inherently limited, but generation-style abstractive methods have proven challenging to build. In this work, we propose a fully data-driven approach to abstractive sentence summarization. Our method utilizes a local attention-based model that generates each word of the summary conditioned on the input sentence. While the model is structurally simple, it can easily be trained end-to-end and scales to a large amount of training data. The model shows significant performance gains on the DUC-2004 shared task compared with several strong baselines.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Convolutional Sequence to Sequence Learning (2017, methodology): “We also evaluate on a Gigaword test set of 2000 pairs which is identical to the one used by Rush et al. (2015) and we report F1 ROUGE similar to prior work.”
Exploring the Limits of Language Modeling (2016, neutral): “, or text summarization (Rush et al., 2015; Filippova et al., 2015).”
Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, background): “…neural network approaches to natural language understanding tasks like translation (Sutskever et al., 2014; Cho et al., 2014), summarization (Rush et al., 2015), and classiﬁcation (Yang et al., 2016) depend crucially on subsystems called sentence encoders that construct distributed…”
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “…weather to be the coolest and fun day and he went to the beach and got ready. primarily from the news domain, abstractive summarization has been approached as a supervised neural sequence transduction problem (Rush et al., 2015; Nallapati et al., 2016; Narayan et al., 2018; Fabbri et al., 2019).”
Generating Wikipedia by Summarizing Long Sequences (2018, methodology): “More recently, neural techniques have been applied to do single-document, abstractive (paraphrasing) text summarization of news articles (Rush et al. (2015), Nallapati et al. (2016)).”
Sequence-Level Knowledge Distillation (2016, background): “, 2015a), summarization (Rush et al., 2015), dialogue (Vinyals and Le, 2015; Serban et al.”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “Commonsense reasoning (Levesque et al., 2012) and abstractive summarization (Rush et al., 2015) represent two areas where these challenges remain readily apparent (Kry ´ sci ´ nski et al., 2019).”
Unified Language Model Pre-training for Natural Language Understanding and Generation (2019, methodology): “We use the non-anonymized version of the CNN/DailyMail dataset [37] and Gigaword [36] for model fine-tuning and evaluation.”
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (2018, background): “NMT has shown remarkable re-sults in several shared tasks (Denkowski and Neu-big, 2017; Nakazawa et al., 2017), and its effective approach has had a strong inﬂuence on other re-lated NLP tasks such as dialog generation (Vinyals and Le, 2015) and automatic summarization (Rush et al., 2015).”
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (2018, methodology): “Promising avenues for future work are to apply subword regularization to other NLP tasks based on encoder-decoder architectures, e.g., dialog generation (Vinyals and Le, 2015) and automatic summarization (Rush et al., 2015).”
Get To The Point: Summarization with Pointer-Generator Networks (2017, background): “However, the recent success of sequence-to-sequence models (Sutskever et al., 2014), in which recurrent neural networks (RNNs) both read and freely generate text, has made abstractive summarization viable (Chopra et al., 2016; Nallapati et al., 2016; Rush et al., 2015; Zeng et al., 2016).”
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond (2016, methodology): “Comparison with state-of-the-art: (Rush et al., 2015) reported recall-only from full-length version of Rouge, but the authors kindly provided us with their F1 numbers, as well as”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “…configurations, Graham (2015) find that none of them outperformed BLEU and that best performance was achieved with the precision vari- 6Note though that DUC 2005 evaluated query-focused summarization instead of sentence compression which was the task studied by Rush et al. (2015). ant of ROUGE-2.”
WikiLingua: A New Benchmark Dataset for Multilingual Abstractive Summarization (2020, methodology): “Rush et al. (2015) proposes the ﬁrst neural ab-stractive summarization model using an attention-based convolutional neural network encoder and a feed-forward decoder.”
MLSUM: The Multilingual Summarization Corpus (2020, methodology): “Gigaword Again using newswire as source data, the english Gigaword (Napoles et al., 2012; Rush et al., 2015; Chopra et al., 2016) corpus is characterized by its large size and the high diversity in terms of sources.”
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, methodology): “…text summarization and evaluate on 12 downstream datasets spanning news (Hermann et al., 2015; Narayan et al., 2018; Grusky et al., 2018; Rush et al., 2015; Fabbri et al., 2019), science (Cohan et al., 2018), short stories (Kim et al., 2019), instructions (Koupaee & Wang, 2018), emails…”
Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, background): “Unlike most existing abstractive approaches (Rush et al., 2015; Chen et al., 2016; Nallapati et al., 2016; See et al., 2017; Tan and Wan, 2017; Paulus et al., 2018; Pasunuru and Bansal, 2018; Celikyilmaz et al., 2018) which rely on an encoder-decoder architecture modeled by recurrent neural…”
Learning to summarize from human feedback (2020, background): “To make short-term progress towards this goal, we focus on abstractive English text summarization, as it has a long history in the NLP community [16, 8, 54, 59, 50], and is a subjective task where we believe it is difficult to quantify summary quality without human judgments.”