Factual Error Correction for Abstractive Summarization Models

By Mengyao Cao, Yue Dong, Jiapeng Wu, Jackie Chi Kit Cheung. 2020

189 citations recorded by the source corpus.

Neural abstractive summarization systems have achieved promising progress, thanks to the availability of large-scale datasets and models pre-trained with self-supervised methods. However, ensuring the factual consistency of the generated summaries for abstractive summarization systems is a challenge. We propose a post-editing corrector module to address this issue by identifying and correcting factual errors in generated summaries. The neural corrector model is pre-trained on artificial examples that are created by applying a series of heuristic transformations on reference summaries. These transformations are inspired by an error analysis of state-of-the-art summarization model outputs. Experimental results show that our model is able to correct factual errors in summaries generated by other neural summarization models and outperforms previous models on factual consistency evaluation on the CNN/DailyMail dataset. We also find that transferring from artificial error correction to downstream settings is still very challenging.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Many of these metrics rely on artificially introduced errors, but Cao et al. (2020) find that moving from artificial to real error detection is challenging, an issue that Zeng et al. (2021) aim to address by using adversarial examples instead.”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “, 2020) and semi-supervised methods to correct factual errors (Cao et al., 2020; Lee et al., 2022a; Balachandran et al., 2022).”
Survey of Hallucination in Natural Language Generation (2022, background): “[14] propose a post-editing corrector module to address this issue by identifying and correcting factual errors in generated summaries.”
GPTScore: Evaluate as You Desire (2023, background): “…example, DEAM (Ghazarian et al., 2022) and Quan-tiDCE (Ye et al., 2021) were proposed for the evaluation of the coherence of the dialogue system; several evaluators (Cao et al., 2020; Durmus et al., 2020; Wang et al., 2020a) are designed for the evaluation of the consistency of text summarization.”
Gradient-Based Adversarial Factual Consistency Evaluation for Abstractive Summarization (2021, methodology): “We follow the recent methods to generate inconsistent samples through rule-based transformations (Kryscinski et al., 2020; Cao et al., 2020).”
Evidence-based Factual Error Correction (2020, methodology): “Cao et al. (2020) generate corrections as a post-editing step for outputs from abstrac-tive summarization so that they are consistent with the source text.”
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations (2022, unknown): “This factuality information at subsentence-level can provide deeper insights on the kinds of factual inconsistencies made by different summarization models (Maynez et al., 2020) and can supply text generation approaches with localized signals for training (Cao et al., 2020; Zhu et al., 2021).”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, unknown): “Another common practice involves constructing an auxiliary fixer to rectify hallucinations during the post-processing stage (Cao et al., 2020; Zhu et al., 2021; Fabbri et al., 2022).”
Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization (2021, background): “Cao et al. (2020) induce artificial perturbations in text to train a summary error correction system, but find that there is a large gap between such artificial perturbations and the type of hallucinations that are generated by abstractive summa-”
Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection (2021, methodology): “A concurrent study to this work (Cao et al., 2020) uses similar strategies as in this paper on a dataset with a very small fraction of hallucinations present.”
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization (2022, background): “The ﬁrst approach proposes post-processing models, that either removes hallucinations in the generated summaries (Cao et al., 2020; Dong et al., 2020), or selects the most factual candidate during beam search (Chen et al., 2021).”
Factual Error Correction for Abstractive Summaries Using Entity Retrieval (2022, unknown): “For baseline seq2seq model, we use bart-base 3 following the previous work (Cao et al., 2020) and train the model using the same dataset we used for training RFEC with same epochs for fair comparison.”
Annotating and Modeling Fine-grained Factuality in Summarization (2021, methodology): “Localizing errors potentially allows for post-hoc correction (Zhao et al., 2020; Cao et al., 2020); however, repairing a summary to be fully factual is a very hard problem and past work has focused on a subset of errors as a result.”
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization (2021, methodology): “First, a separate correction model is learned to ﬁx errors made by the summarizers (Zhao et al., 2020; Chen et al., 2021), including replacing entities absent from the source (Dong et al., 2020) or revising all possible errors (Cao et al., 2020).”
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling (2022, methodology): “We see the largest improvements in the Semantic Frame Error category with an increase of ∼ 8 points on CNN/DM and ∼ 13 points on XSum.”
Multi-Fact Correction in Abstractive Text Summarization (2020, background): “The most similar work to ours is proposed concurrently by Meng et al. (2020), where they ﬁne-tune a BART (Lewis et al., 2020)”
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries (2021, background): “Thus, evaluating the factual consistency of the generated summaries with respect to the source is an important task (Falke et al., 2019; Cao et al., 2020; Gabriel et al., 2021; Durmus et al., 2020; Huang et al., 2021; Pagnoni et al., 2021).”
Enhancing Factual Consistency of Abstractive Summarization (2021, methodology): “Concurrent to our work, Cao et al. (2020) employs the generation model BART (Lewis et al., 2019) to produce corrected summaries.”