Factual Consistency Evaluation for Text Summarization via Counterfactual Estimation

By Yuexiang Xie, Fei Sun, Yang Deng, Yaliang Li, Bolin Ding. 2021

62 citations recorded by the source corpus.

Despite significant progress has been achieved in text summarization, factual inconsistency in generated summaries still severely limits its practical applications. Among the key factors to ensure factual consistency, a reliable automatic evaluation metric is the first and the most crucial one. However, existing metrics either neglect the intrinsic cause of the factual inconsistency or rely on auxiliary tasks, leading to an unsatisfied correlation with human judgments or increasing the inconvenience of usage in practice. In light of these challenges, we propose a novel metric to evaluate the factual consistency in text summarization via counterfactual estimation, which formulates the causal relationship among the source document, the generated summary, and the language prior. We remove the effect of language prior, which can cause factual inconsistency, from the total causal effect on the generated summary, and provides a simple yet effective way to evaluate consistency without relying on other auxiliary tasks. We conduct a series of experiments on three public abstractive text summarization datasets, and demonstrate the advantages of the proposed metric in both improving the correlation with human judgments and the convenience of usage. The source code is available at https://github.com/xieyxclack/factual_coco.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Instantiations of these metrics may blank out entities (Eyal et al., 2019; Xie et al., 2021; Scialom et al., 2019), or fully 5ROUGE is a special case of this where basic elements are fixed size n-grams, but other basic element metrics like PARENT (Dhingra et al., 2019) only focus on content…”
TRUE: Re-evaluating Factual Consistency Evaluation (2022, methodology): “…and methods that target a speciﬁc task or domain, such as summarization (Falke et al., 2019; Kryscinski et al., 2020; Wang et al., 2020; Scialom et al., 2021; Deutsch et al., 2021; Xie et al., 2021) or dialogue (Dziri et al., 2021; Honovich et al., 2021; Nie et al., 2021; Qin et al., 2021).”
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations (2022, unknown): “Xie et al. (2021) formulate causal relationships among the document, summary, and language prior to evaluate the factu-ality via counterfactual estimation.”