Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection

By Sihao Chen, Fan Zhang, Kazoo Sone, Dan Roth. 2021

119 citations recorded by the source corpus.

Despite significant progress in neural abstractive summarization, recent studies have shown that the current models are prone to generating summaries that are unfaithful to the original context. To address the issue, we study contrast candidate generation and selection as a model-agnostic post-processing technique to correct the extrinsic hallucinations (i.e. information not present in the source text) in unfaithful summaries. We learn a discriminative correction model by generating alternative candidate summaries where named entities and quantities in the generated summary are replaced with ones with compatible semantic types from the source document. This model is then used to select the best candidate as the final output summary. Our experiments and analysis across a number of neural summarization systems show that our proposed method is effective in identifying and correcting extrinsic hallucinations. We analyze the typical hallucination phenomenon by different types of neural summarization systems, in hope to provide insights for future work on the direction.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, methodology): “Examples of the augmented information include entity information [114], extracted relation triples from source document [19, 74] obtained by Fact Description Extraction, pre-executed operation results [136], synthetic data generated through replacement or perturbation [21, 95], retrieved external kn”
Factuality Enhanced Language Models for Open-Ended Text Generation (2022, background): “Hallucination in downstream NLG tasks There are active efforts to reduce the unfaithfulness or factual errors of task-speciﬁc LMs ﬁne-tuned for various downstream natural language generation (NLG) tasks such as summarization [42–48], data-to-text [49, 50, 20, 51–53] and dialogue system [54–58].”
Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization (2021, methodology): “…is important to the study. world scenarios, as such recent work has studied methods to improve the faithfulness of abstractive summarization systems (Matsumaru et al., 2020; Zhao et al., 2020; Dong et al., 2020; Goyal and Durrett, 2021; Xu et al., 2020; Chen et al., 2021; Zhu et al., 2021).”
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization (2021, methodology): “First, a separate correction model is learned to ﬁx errors made by the summarizers (Zhao et al., 2020; Chen et al., 2021), including replacing entities absent from the source (Dong et al., 2020) or revising all possible errors (Cao et al., 2020).”
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization (2022, methodology): “The negative relationship between factuality metrics and ROUGE is also reported in prior works (Chen et al., 2021; Kryscinski et al., 2019).”
Improving Factual Consistency of Abstractive Summarization via Question Answering (2021, methodology): “Another recent work tackles the hallucination problem in abstractive text summarization via post processing on the generated summary (Chen et al., 2021).”