Multi-Fact Correction in Abstractive Text Summarization

By Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, Jingjing Liu. 2020

133 citations recorded by the source corpus.

Pre-trained neural abstractive summarization systems have dominated extractive strategies on news summarization performance, at least in terms of ROUGE. However, system-generated abstractive summaries often face the pitfall of factual inconsistency: generating incorrect facts with respect to the source text. To address this challenge, we propose Span-Fact, a suite of two factual correction models that leverages knowledge learned from question answering models to make corrections in system-generated summaries via span selection. Our models employ single or multi-masking strategies to either iteratively or auto-regressively replace entities in order to ensure semantic consistency w.r.t. the source text, while retaining the syntactic structure of summaries generated by abstractive summarization models. Experiments show that our models significantly boost the factual consistency of system-generated summaries without sacrificing summary quality in terms of both automatic metrics and human evaluation.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, methodology): “[34] use the source input as the “fact” to determine the factual correctness, making "factuality" indistinguishable from "faithfulness".”
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization (2021, result): “If a model can pinpoint which portion of a summary is inconsistent, some work has shown that corrector models can effectively re-write the problematic portions and often remove the inconsistency (Dong et al., 2020).”
Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization (2021, background): “(2020); Dong et al. (2020); Durmus et al. (2020) measure and improve the factual consistency of summaries by asking and answering questions based on generated summaries”
Annotating and Modeling Fine-grained Factuality in Summarization (2021, methodology): “Recent work has explored different methods for enforcing factuality: modifying the model, such as encoding SRL structures in the input (Cao et al., 2018), post-hoc correction (Dong et al., 2020), or constrained decoding (Song et al., 2020a; Mao et al., 2020).”
The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey (2021, unknown): “To train SpanFact, [ Dong et al. , 2020 ] construct the dataset automatically.”
Generating SOAP Notes from Doctor-Patient Conversations Using Modular Summarization Techniques (2020, background): “The detection and correction of such factual errors in automatically generated summaries is an active area of research (Cao et al., 2018; Zhang et al., 2020; Dong et al., 2020).”
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization (2022, methodology): “To tackle this problem, many approaches propose post-processing models (Chen et al., 2021; Dong et al., 2020; Liu and Liu, 2021), but such methods are often constrained by external resources to train additional correction or selection models.”
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling (2022, methodology): “We construct factually incorrect summaries by replacing the mask with the lower ranked candidates.”
Constrained Abstractive Summarization: Preserving Factual Consistency with Constrained Generation (2020, unknown): “Human evaluations of similar sizes (50/100/109/10 samples) are conducted by previous studies (Dong et al., 2020; Zhu et al., 2021; Matsumaru et al., 2020; Nan et al., 2021).”
Enhancing Factual Consistency of Abstractive Summarization (2021, methodology): “To correct factual errors, Dong et al. (2020) uses pre-trained NLU models to rectify one or more wrong entities in the summary.”
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding (2023, background): “…(Cao et al., 2018; Zhu et al., 2021), optimizing summarization models towards a factual consistency metrics (Nan et al., 2021; Cao and Wang, 2021), learning a post-editing error corrector (Dong et al., 2020) and removing noisy training samples (Kang and Hashimoto, 2020; Goyal and Durrett, 2021).”
On Generating Extended Summaries of Long Documents (2020, background): “…Zhai, and Zhou 2017; Zhou et al. 2018; Liu and Lapata 2019; Xu et al. 2020; Jia et al. 2020) and abstractive (e.g., See, Liu, and Manning 2017; Cohan et al. 2018; MacAvaney et al. 2019; Zhang et al. 2019; Sotudeh, Goharian, and Filice 2020; Dong et al. 2020) approaches for document summarization.”
Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization (2021, methodology): “…is important to the study. world scenarios, as such recent work has studied methods to improve the faithfulness of abstractive summarization systems (Matsumaru et al., 2020; Zhao et al., 2020; Dong et al., 2020; Goyal and Durrett, 2021; Xu et al., 2020; Chen et al., 2021; Zhu et al., 2021).”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “Furthermore, in text summarization, some studies proposed evaluation metrics or correction methods to measure and revise the generated text for preserving factuality [46, 172].”
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization (2021, methodology): “Three types of remedies have been proposed: running a separately learned error correction component (Dong et al., 2020), removing noisy training samples (Nan et al., 2021; Goyal and Durrett, 2021), and modifying the Transformer architecture (Huang et al., 2020; Zhu et al., 2021).”
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation (2021, methodology): “To reduce hallucinations in the reference-based setting, researchers have applied iterative training (Nie et al., 2019), post editing (Dong et al., 2020), 13https://www.statcan.gc.ca/en/ statistical-programs/instrument/3438_ Q1_V6 soft constraints, e.g. attention manipulation (Kiddon et al., 2016;…”
Gradient-Based Adversarial Factual Consistency Evaluation for Abstractive Summarization (2021, background): “(2) Use natural language inference or question answering models for fact checking correction (Li et al., 2018; Falke et al., 2019; Wang et al., 2020; Dong et al., 2020; Durmus et al., 2020; Chen et al., 2020).”
Don’t Say What You Don’t Know: Improving the Consistency of Abstractive Summarization by Constraining Beam Search (2022, methodology): “Prior works attempt to improve consistency by correcting already-generated summaries (Dong et al., 2020; Zhu et al., 2021), using a knowledge graph (Zhu et al.”