Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting

By Yen-Chun Chen, Mohit Bansal. 2018

610 citations recorded by the source corpus.

Inspired by how humans summarize long documents, we propose an accurate and fast summarization model that first selects salient sentences and then rewrites them abstractively (i.e., compresses and paraphrases) to generate a concise overall summary. We use a novel sentence-level policy gradient method to bridge the non-differentiable computation between these two neural networks in a hierarchical way, while maintaining language fluency. Empirically, we achieve the new state-of-the-art on all metrics (including human evaluation) on the CNN/Daily Mail dataset, as well as significantly higher abstractiveness scores. Moreover, by first operating at the sentence-level and then the word-level, we enable parallel decoding of our neural generative model that results in substantially faster (10-20x) inference speed as well as 4x faster training convergence than previous long-paragraph encoder-decoder models. We also demonstrate the generalization of our model on the test-only DUC-2002 dataset, where we achieve higher scores than a state-of-the-art model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

WikiLingua: A New Benchmark Dataset for Multilingual Abstractive Summarization (2020, background): “Chen and Bansal (2018) takes a two stage approach to abstractive summarization by learning an extractor to select salient sentences from the articles, and an abstractor to rewrite the sentences selected by the extractor.”
Better Rewards Yield Better Summaries: Learning to Summarise Without References (2019, methodology): “We consider two RL-based summarisation systems, an extractive system NeuralTD (Gao et al., 2019), and an abstractive system ExtAbsRL (Chen and Bansal, 2018).”
On Faithfulness and Factuality in Abstractive Summarization (2020, background): “Recently, Chen and Bansal (2018) asked human annotators to assess the summary relevance measuring both the saliency and the presence of contradictory/unrelated information.”
SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization (2019, methodology): “This requires natural language generation techniques with high level of semantic understanding (Chopra et al., 2016; Rush et al., 2015; Khandelwal et al., 2019; Zhang et al., 2019; See et al., 2017; Chen and Bansal, 2018; Gehrmann et al., 2018).”
Evaluating the Factual Consistency of Abstractive Text Summarization (2019, methodology): “nsists of 931 examples, the test set contains 503 examples. The model outputs used for annotation were provided by the authors of papers:Hsu et al.(2018);Gehrmann et al. (2018);Jiang and Bansal(2018);Chen and Bansal (2018);See et al.(2017);Kry´sci nski et al.´ (2018); Li et al.(2018b);Pasunuru and B”
Neural Text Summarization: A Critical Evaluation (2019, methodology): “…Pasunuru and Bansal, 2018; Kry´sci´nski et al., 2018), reinforcement learning strategies (Paulus et al., 2017; Narayan et al., 2018b; Dong et al., 2018; Wu and Hu, 2018), and hybrid extractive-abstractive models (Liu et al., 2018; Hsu et al., 2018; Gehrmann et al., 2018; Chen and Bansal, 2018).”
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries (2020, methodology): “The experiment uses 373 triplets of source sentences from CNN/DM and two summary sentences generated from the model from Chen and Bansal (2018).”
BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization (2019, methodology): “For RNN-EXT RL from Chen and Bansal (2018), we used the implementation provided by the authors 9 .”
This Email Could Save Your Life: Introducing the Task of Email Subject Line Generation (2019, methodology): “Based on recent progress in news summarization (Chen and Bansal, 2018), our model generates email subjects in two stages: (1) The extractor selects multiple sentences containing salient information for writing a subject ( § 3.1).”
BanditSum: Extractive Summarization as a Contextual Bandit (2018, methodology): “…Wu and Hu (2018) use reinforcement learning in a sequential labeling setting to train ab-stractive and extractive summarizers, respectively, while Chen and Bansal (2018) combines both approaches, applying abstractive summarization to a set of sentences extracted by a pointer network (Vinyals et…”
Answers Unite! Unsupervised Metrics for Reinforced Summarization Models (2019, neutral): “10 Chen and Bansal (2018) 40.”
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020, background): “Neural abstractive models are effective at identifying salient content and producing ﬂuent summaries (See et al., 2017; Chen and Bansal, 2018; Gehrmann et al., 2018).”
Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference (2019, methodology): “Incorrect FAS The hybrid extractive-abstractive system proposed by Chen and Bansal (2018) including their redundancy-based reranking.”
On Extractive and Abstractive Neural Document Summarization with Transformer Language Models (2020, methodology): “Chen and Bansal (2018) use a pointer network (Vinyals et al., 2015) to sequentially pick sentences from the document that comprise its extractive summary.”
Evaluation of Text Generation: A Survey (2020, methodology): “…the issues found in BLEU and has been widely used for evaluating machine translation models and other models, such as image captioning (Kilickaya et al., 2017), question generation (Nema & Khapra, 2018; Du et al., 2017), and summarization (See et al., 2017; Chen & Bansal, 2018; Yan et al., 2020).”
Sample Efficient Text Summarization Using a Single Pre-Trained Transformer (2019, methodology): “, 2017) or reinforcement learning (Chen and Bansal, 2018).”
BARTScore: Evaluating Generated Text as Text Generation (2021, neutral): “Abstractive Systems A1: Ptr-Gen [62]; A2: Bottom-up [17]; A3: Fast-Abs-RL [5]; A4: Two-stage-RL [74]; A5: BERT-Ext-Abs [41]; A6: BERT-Abs [41]; A7: Trans-Abs [41]; A8: UniLM-1 [10]; A9: UniLM-2 [2]; A10: T5-base [56]; A11: T5-large [56]; A12: T5-11B [56]; A13: BART [33]; A14: SemSim [42].”
HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization (2019, background): “SentRewrite (Hsu et al., 2018) and InconsisLoss (Chen and Bansal, 2018) all try to decompose the word by word summary generation into sentence selection from document and “sentence” level summarization (or compression).”
An Overview of Natural Language State Representation for Reinforcement Learning (2020, methodology): “Wu & Hu (2018) built a document encoding with a CNN operating at word level and a BiGRU at sentence level, whereas Narayan et al. (2018) and Chen & Bansal (2018) used a CNN at sentence level and an LSTM or BiLSTM at document level to capture global information.”