Get To The Point: Summarization with Pointer-Generator Networks

By Abigail See, Peter J. Liu, Christopher D. Manning. 2017

4,463 citations recorded by the source corpus.

Neural sequence-to-sequence models have provided a viable new approach for abstractive text summarization (meaning they are not restricted to simply selecting and rearranging passages from the original text). However, these models have two shortcomings: they are liable to reproduce factual details inaccurately, and they tend to repeat themselves. In this work we propose a novel architecture that augments the standard sequence-to-sequence attentional model in two orthogonal ways. First, we use a hybrid pointer-generator network that can copy words from the source text via pointing, which aids accurate reproduction of information, while retaining the ability to produce novel words through the generator. Second, we use coverage to keep track of what has been summarized, which discourages repetition. We apply our model to the CNN / Daily Mail summarization task, outperforming the current abstractive state-of-the-art by at least 2 ROUGE points.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Deep Reinforced Model for Abstractive Summarization (2017, methodology): “In parallel of our work, See et al. (2017) also developed an abstractive summarization model on this dataset with an extra loss term to increase temporal coverage of the encoder attention function.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “It has also been argued that generative models trained with maximum likelihood are prone to producing repetitive summaries [See et al., 2017].”
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, background): “(supervised) summarization task (See et al., 2017) and whether abstractive models provide much of a benefit over extractive ones on it (Kryściński et al.”
Fine-Tuning Language Models from Human Preferences (2019, methodology): “This copying behavior emerged naturally from the data collection and training process; we did not use any explicit architectural mechanism for copying as in See et al. (2017); Gehrmann et al. (2018).”
CoQA: A Conversational Question Answering Challenge (2018, methodology): “Additionally, as the answer words are likely to appear in the original passage, we employ a copy mechanism in the decoder which allows to (optionally) copy a word from the passage (Gu et al., 2016; See et al., 2017).”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “Coverage mechanisms (Nallapati et al., 2016; See et al., 2017; Suzuki and Nagata, 2017) and temporal attention (Paulus et al.”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “The problem of repetition can instead be approached by altering the training objectives, as with coverage mechanisms (See et al., 2017) and context-based losses (Welleck et al., 2019).”
Unified Language Model Pre-training for Natural Language Understanding and Generation (2019, methodology): “• PGNet (See et al., 2017): a sequence-to-sequence model based on the pointer-generator network.”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, methodology): “the full-text version with no entity anonymization (See et al., 2017); we truncate articles to 400 tokens (See et al.”
Scaling Neural Machine Translation (2018, methodology): “Similar sequence-to-sequence models are also applied to other natural language processing tasks, such as abstractive summarization (See et al., 2017; Paulus et al., 2018) and dialog (Sordoni et al.”
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, neutral): “, 2018), CNN/Dailymail (See et al., 2017) and MultiNews (Fabbri et al.”
Finetuned Language Models Are Zero-Shot Learners (2021, neutral): “CNN-DM (See et al., 2017) 3.”
WikiLingua: A New Benchmark Dataset for Multilingual Abstractive Summarization (2020, methodology): “See et al. (2017) extends this model by further incorporating a coverage penalty to address the problem of repetitions in the generated summary.”
Graph-based, Self-Supervised Program Repair from Diagnostic Feedback (2020, background): “…the probability of a line k ∈ { 1 , . . . , L } being erroneous via a feedforward network, and model its repair y k , via a pointer-generator decoder (See et al., 2017): Expected ... ● operator/punctuator ● primary expression expected @@@ (e.g. expected ‘;’ before..., expected ‘}’ at end of input,…”
MLSUM: The Multilingual Summarization Corpus (2020, methodology): “Over the last few years, advances in deep learning applied to NLP have contributed to the rising popularity of this task among the research community (See et al., 2017; Kry´sci´nski et al., 2018; Scialom et al., 2019).”
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, methodology): “Most prior work on neural abstractive summarization relied on large-scale, high-quality datasets of supervised document-summary pairs (Hermann et al., 2015) and achieved promising results (Rush et al., 2015; Nallapati et al., 2016; See et al., 2017).”
Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, methodology): “…single-document summarization has consistently attracted attention (Cheng and Lapata, 2016; Durrett et al., 2016; Nallapati et al., 2016, 2017; See et al., 2017; Tan and Wan, 2017; Narayan et al., 2017; Fan et al., 2017; Paulus et al., 2018; Pasunuru and Bansal, 2018; Celikyilmaz et al.,…”