Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond

By Ramesh Nallapati, Bowen Zhou, C. D. Santos, Caglar Gulcehre, Bing Xiang. 2016

2,864 citations recorded by the source corpus.

In this work, we model abstractive text summarization using Attentional Encoder-Decoder Recurrent Neural Networks, and show that they achieve state-of-the-art performance on two different corpora. We propose several novel models that address critical problems in summarization that are not adequately modeled by the basic architecture, such as modeling key-words, capturing the hierarchy of sentence-to-word structure, and emitting words that are rare or unseen at training time. Our work shows that many of our proposed models contribute to further improvement in performance. We also propose a new dataset consisting of multi-sentence summaries, and establish performance benchmarks for further research.

Subfields: Artificial Intelligence, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Generative Models

Citations in the corpus

Convolutional Sequence to Sequence Learning (2017, background): “Sequence to sequence learning has been successful in many tasks such as machine translation, speech recognition (Sutskever et al., 2014; Chorowski et al., 2015) and text summarization (Rush et al., 2015; Nallapati et al., 2016; Shen et al., 2016) amongst others.”
Learned in Translation: Contextualized Word Vectors (2017, background): “Recent work in NLP has continued in this direction by using pretrained word representations to improve models for entailment [Bowman et al., 2014], sentiment analysis [Socher et al., 2013], summarization [Nallapati et al., 2016], and question answering [Seo et al., 2017, Xiong et al., 2017].”
Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “…the performance of PPO and DPO under distribution shifts, we evaluate the PPO and DPO policies from our Reddit TL;DR summarization experiment on a different distribution, news articles in the test split of the CNN/DailyMail dataset [24], using the best sampling temperatures from TL;DR (0 and 0.25).”
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, background): “…weather to be the coolest and fun day and he went to the beach and got ready. primarily from the news domain, abstractive summarization has been approached as a supervised neural sequence transduction problem (Rush et al., 2015; Nallapati et al., 2016; Narayan et al., 2018; Fabbri et al., 2019).”
The Curious Case of Neural Text Degeneration (2019, background): “Example applications include machine translation (Bahdanau et al., 2015), data-to-text generation (Wiseman et al., 2017), and summarization (Nallapati et al., 2016).”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “Summarization Recent approaches combine recurrent neural networks with pointer networks to generate output sequences that contain key words copied from the document (Nallapati et al., 2016).”
Generating Wikipedia by Summarizing Long Sequences (2018, methodology): “More recently, neural techniques have been applied to do single-document, abstractive (paraphrasing) text summarization of news articles (Rush et al. (2015), Nallapati et al. (2016)).”
Get To The Point: Summarization with Pointer-Generator Networks (2017, methodology): “However, the recent success of sequence-to-sequence models (Sutskever et al., 2014), in which recurrent neural networks (RNNs) both read and freely generate text, has made abstractive summarization viable (Chopra et al., 2016; Nallapati et al., 2016; Rush et al., 2015; Zeng et al., 2016).”
Unsupervised Pretraining for Sequence to Sequence Learning (2016, methodology): “Following Nallapati et al. (2016), we modify the data collection scripts to restore the bullet point summaries.”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, methodology): “We use the CNN-Dailymail dataset (Hermann et al., 2015; Nallapati et al., 2016) of news articles paired with multi-sentence summaries.”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…al. (2019) 5- way annotated system outputs on 100 samples from the test set of the CNN-Dailymail summarization corpus (CNNDM, Hermann et al., 2015; Nallapati et al., 2016) along two measures of content quality (relevance of the content, and faithfulness) and two of linguistic quality (on the…”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, background): “, 2011) and the CNN/DailyMail summarization dataset (Hermann et al., 2015; Nallapati et al., 2016), or it is uploaded after creation (e.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “To demonstrate this trade-off, consider an improvement on the CNN-DM summarization dataset (Hermann et al., 2015; Nallapati et al., 2016) measured by the ROUGE-L met-”
WikiLingua: A New Benchmark Dataset for Multilingual Abstractive Summarization (2020, background): “Nallapati et al. (2016) shows further improvements by incorporating embeddings for linguistic features such as part-of-speech tags and named-entity tags into their model, as well as a pointer network (Vinyals et al., 2015) to enable copying words from the source article.”
MLSUM: The Multilingual Summarization Corpus (2020, methodology): “CNN / Daily Mail One of the most commonly used dataset for summarization (Nallapati et al., 2016; See et al., 2017; Paulus et al., 2017; Dong et al., 2019), although originally built for Question Answering tasks (Hermann et al., 2015a).”