A Deep Reinforced Model for Abstractive Summarization

By Romain Paulus, Caiming Xiong, Richard Socher. 2017

1,668 citations recorded by the source corpus.

Attentional, RNN-based encoder-decoder models for abstractive summarization have achieved good performance on short input and output sequences. For longer documents and summaries however these models often include repetitive and incoherent phrases. We introduce a neural network model with a novel intra-attention that attends over the input and continuously generated output separately, and a new training method that combines standard supervised word prediction and reinforcement learning (RL). Models trained only with supervised learning often exhibit "exposure bias" - they assume ground truth is provided at each step during training. However, when standard word prediction is combined with the global sequence prediction training of RL the resulting summaries become more readable. We evaluate this model on the CNN/Daily Mail and New York Times datasets. Our model obtains a 41.16 ROUGE-1 score on the CNN/Daily Mail dataset, an improvement over previous state-of-the-art models. Human evaluation also shows that our model produces higher quality summaries.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Attention is All you Need (2017, background): “Self-attention has been used successfully in a variety of tasks including reading comprehension, abstractive summarization, textual entailment and learning task-independent sentence representations [4, 22, 23, 19].”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “It has been shown that improvements to the ROUGE score do not necessarily correspond to more coherent summaries [Paulus et al., 2017].”
Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, background): “These methods represent a convergence of two bodies of work: one body of work on training language models with reinforcement learning for a variety of objectives [33, 27, 45] and another body of work on general methods for learning from human preferences [12, 19].”
Fine-Tuning Language Models from Human Preferences (2019, background): “On the capability side, purely supervised training is insufficient to correct mistakes that arise when sampling from trained policies, and RL training to programmatic reward functions such as BLEU or ROUGE is insufficient: Paulus et al. (2017) conclude that "optimizing for single discrete evaluation”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “Reinforcement learning has pushed performance using common summarization metrics [Paulus et al., 2017] as well as alternative metrics that transfer knowledge from another task [Pasunuru et al., 2017, Pasunuru and Bansal, 2018].”
Generating Wikipedia by Summarizing Long Sequences (2018, methodology): “Although optimizing ROUGE directly has been shown to not always yield the best summaries as evaluated by human judgment (Paulus et al., 2017), we found that for our task optimizing for perplexity correlates with increased ROUGE and human judgment.”
Unified Language Model Pre-training for Natural Language Understanding and Generation (2019, methodology): “We remove duplicated trigrams in beam search, and tweak the maximum summary length on the development set (Paulus et al., 2018; Edunov et al., 2019).”
Fine-tune BERT for Extractive Summarization (2019, methodology): “Compared baselines are (1) First-k words, which is a simple baseline by extracting ﬁrst k words of the input article; (2) Full is the best-performed extractive model in Dur-rett et al. (2016); (3) Deep Reinforced (Paulus et al., 2018) is an abstractive model, using reinforce learning and encoder-dec”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, methodology): “To generate summaries, we follow standard practice in tuning the minimum output length and disallow repeating the same trigram (Paulus et al., 2017).”
Scaling Neural Machine Translation (2018, methodology): “Similar sequence-to-sequence models are also applied to other natural language processing tasks, such as abstractive summarization (See et al., 2017; Paulus et al., 2018) and dialog (Sordoni et al., 2015; Serban et al., 2017; Dusek and Jurc´ıcek, 2016).”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “For example, Paulus et al. (2018) demonstrated that we may be able to optimize ROUGE directly and Pasunuru and Bansal (2018) explored alternative optimization targets such as maximizing entailment.”
MLSUM: The Multilingual Summarization Corpus (2020, methodology): “Although (Grusky et al., 2018) found indications of bias towards extractive approaches, several research efforts have used this dataset for summarization (Hong and Nenkova, 2014; Durrett et al., 2016; Paulus et al., 2017).”
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, methodology): “…2019), over-penalizing abstractive approaches com- pared to extractive ones and having no sense of linguistic quality, we found that choosing perplexity-optimized models using aggregated ROUGE (rather than directly optimizing ROUGE as in Paulus et al. (2017)) resulted in qualitatively good models.”
Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, methodology): “…has consistently attracted attention (Cheng and Lapata, 2016; Durrett et al., 2016; Nallapati et al., 2016, 2017; See et al., 2017; Tan and Wan, 2017; Narayan et al., 2017; Fan et al., 2017; Paulus et al., 2018; Pasunuru and Bansal, 2018; Celikyilmaz et al., 2018; Narayan et al., 2018a,b).”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “At decoding time, the models use beam search with a beam size of 10 , context and label 3 -gram blocking (Paulus et al., 2017), and a minimum beam length of 20 BPE tokens, shown to perform well in Roller et al. (2020).”
Recipes for Safety in Open-domain Chatbots (2020, methodology): “Decoding At decoding time, the model employs standard beam search with a beam size of 10, context and label 3-gram blocking (Paulus et al., 2017), and a minimum beam length of 20 BPE tokens, which was shown to perform well compared to other choices.”
Learning to summarize from human feedback (2020, methodology): “Indeed, existing automatic metrics for evaluating summary quality, such as ROUGE [39], have received criticism for poor correlation with human judgments [55, 45, 6, 33].”
ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, methodology): “Apart from text GANs, they all suffer from the aforementioned limitations of ill-deﬁned reward metrics, such as BLEU or ROUGE [30].”
Discriminative Adversarial Search for Abstractive Summarization (2020, neutral): “Whether trained from scratch (Paulus et al., 2017; Gehrmann et al., 2018) or based on pre-trained language models (Dong et al., 2019 We report length in tokens for source (len src) and summaries (len tgt).”
How Decoding Strategies Affect the Verifiability of Generated Text (2019, background): “A simple trick to reduce repetitiveness is to explicitly prevent the generation of already observed n-grams (Paulus et al., 2017).”