Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization

By Yiran Chen, Peng Liu, Xipeng Qiu. 2021

29 citations recorded by the source corpus.

With the continuous upgrading of the summarization systems driven by deep neural networks, researchers have higher requirements on the quality of the generated summaries, which should be not only ﬂuent and infor-mative but also factually correct. As a result, the ﬁeld of factual evaluation has developed rapidly recently. Despite its initial progress in evaluating generated summaries, the meta-evaluation methodologies of factuality metrics are limited in their opacity , leading to the insufﬁcient understanding of factuality metrics’ relative advantages and their applicability. In this paper, we present an adversarial meta-evaluation methodology that allows us to (i) diagnose the ﬁne-grained strengths and weaknesses of 6 existing top-performing metrics over 24 diagnostic test datasets, (ii) search for directions for further improvement by data augmentation. Our observations from this work motivate us to propose several calls for future research. We make all codes, diagnostic test datasets, trained factuality models available: https://github.com/zide05/ AdvFact .

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…oftentimes fail at capturing factuality, but identify questionanswering metrics as promising, somewhat contradicting Maynez et al.. Similarly, Chen et al. (2021) analyze mispredictions on a set of previously annotated summarization corpora (Kryscinski et al., 2020; Wang et al., 2020; Falke…”
Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022, methodology): “We use the method in Chen et al. (2021) to construct inconsistent summaries by antonym substitution, numerical editing, entity replacement, and syntactic pruning.”