Improving Abstraction in Text Summarization

By Wojciech Kryscinski, Romain Paulus, Caiming Xiong, Richard Socher. 2018

154 citations recorded by the source corpus.

Abstractive text summarization aims to shorten long text documents into a human readable form that contains the most important facts from the original document. However, the level of actual abstraction as measured by novel phrases that do not appear in the source document remains low in existing approaches. We propose two techniques to improve the level of abstraction of generated summaries. First, we decompose the decoder into a contextual network that retrieves relevant parts of the source document, and a pretrained language model that incorporates prior knowledge about language generation. Second, we propose a novelty metric that is optimized directly through policy learning to encourage the generation of novel phrases. Our model achieves results comparable to state-of-the-art models, as determined by ROUGE scores and human evaluations, while achieving a significantly higher level of abstraction as measured by n-gram overlap with the source document.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

MLSUM: The Multilingual Summarization Corpus (2020, background): “Novelty Because of their use of copy mechanisms, some abstractive models have been reported to rely too much on extraction (See et al., 2017; Kry´sci´nski et al., 2018).”
Discriminative Adversarial Search for Abstractive Summarization (2020, neutral): “Under this framework, an abstractive summarizer is thus trained using article ( X ) and summary ( Y ) pairs (e.g., via log-likelihood maximization).”
Better Rewards Yield Better Summaries: Learning to Summarise Without References (2019, methodology): “Kryscinski et al. (2018) propose a simple heuristic that encourages the RL-based abstractive summariser to generate summaries with more novel tokens, i.e. tokens that do not appear in the input document.”
Evaluating the Factual Consistency of Abstractive Text Summarization (2019, methodology): “2018; Pasunuru and Bansal, 2018), including reinforcement learning techniques (Kryściński et al., 2018; Dong et al., 2018; Wu and Hu, 2018).”
Answers Unite! Unsupervised Metrics for Reinforced Summarization Models (2019, background): “Policy Learning Several RL-based summarization (Kryściński et al., 2018; Li et al., 2018; Pasunuru and Bansal, 2018; Paulus et al., 2017) apply the self-critical policy gradient training algorithm (Rennie et al.”
Reward Learning for Efficient Reinforcement Learning in Extractive Document Summarisation (2019, background): “Recent work suggests that improving the reward function boosts the performance of RL-based NLG [ Kryscinski et al. , 2018 ] .”
Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization (2021, background): “While there has been tremendous progress towards this goal (Kry´sci´nski et al., 2018; Dong et al., 2019; Zhang et al., 2019; Lewis et al., 2020), abstractive summarization systems still suffer from faithfulness errors (Cao et al., 2018), generating information that is not present in the original…”
Abstract Text Summarization: A Low Resource Challenge (2019, unknown): “The abstract text summarization approach aims to shorten the long text into a humanreadable form that contains the most important fact from the original text (Allahyari et al., 2017; Kryściński et al., 2018).”
Concept Pointer Network for Abstractive Summarization (2019, background): “Kryscinski et al. (2018) and Weber et al. (2018) deﬁne the number of new n -grams as the primary criteria of abstractiveness.”
Automatic Construction of Evaluation Suites for Natural Language Generation Datasets (2021, methodology): “To define these subgroups, we use lexical novelty of the reference summaries which has been used as a proxy for abstractiveness in previous works [42, 43, 44].”
Pre-Trained Language Models for Text Generation: A Survey (2022, unknown): “For example, Kryscinski et al. [116] utilized a contextual network in the PLM decoder to retrieve the most salient parts from the source document to improve the level of faithfulness of generated summaries.”
An Overview of Natural Language State Representation for Reinforcement Learning (2020, background): “Paulus et al. (2018), a reference for subsequent works (Kryściński et al., 2018), used context vectors with intra-temporal attention over the hidden states of the encoder (which process the original text) concatenated to the hidden state of the decoder (which generates the summary).”
Controlling the Amount of Verbatim Copying in Abstractive Summarization (2019, background): “To suppress copying, Kryściński et al. (2018) introduce a novelty metric which is to be optimized during policy learning; and Weber et al. (2018) modify the scoring function of the summary sequence at decoding time.”
Joint Parsing and Generation for Abstractive Summarization (2019, background): “Abstractive summarization has made considerable recent progress (See, Liu, and Manning 2017; Chen and Bansal 2018; Kryscinski et al. 2018).”
HighRES: Highlight-based Reference-less Evaluation of Summarization (2019, methodology): “Most recent work uses a single human judgment to capture all linguistic qualities of the summary (Hsu et al., 2018; Kryściński et al., 2018; Narayan et al., 2018b; Song et al., 2018; Guo et al., 2018); we group them under “Fluency” in Table 1 with an exception of “Clarity” which was evaluated in…”