GO FIGURE: A Meta Evaluation of Factuality in Summarization

By Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, Jianfeng Gao. 2020

107 citations recorded by the source corpus.

While neural language models can generate text with remarkable fluency and coherence, controlling for factual correctness in generation remains an open research question. This major discrepancy between the surface-level fluency and the content-level correctness of neural generation has motivated a new line of research that seeks automatic metrics for evaluating the factuality of machine text. In this paper, we introduce GO FIGURE, a meta-evaluation framework for evaluating factuality evaluation metrics. We propose five necessary and intuitive conditions to evaluate factuality metrics on diagnostic factuality data across three different summarization tasks. Our benchmark analysis on ten factuality metrics reveals that our meta-evaluation framework provides a robust and efficient evaluation that is extensible to multiple types of factual consistency and standard generation metrics, including QA metrics. It also reveals that while QA metrics generally improve over standard metrics that measure factuality across domains, performance is highly dependent on the way in which questions are generated.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “In an alternative approach to measuring correlations with human judgments, Gabriel et al. (2021) inject factual errors in reference summaries, and checks whether system rankings produced by metrics correlate with the “level of factuality” of the transformed sentences, among other properties like a…”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “Since ROUGE only tests the extent to which a generated summary has a lexical overlap with a reference summary, it can erroneously produce high scores for fluent, yet meaningless and unfaithful outputs as long as many of the same words are used (Maynez et al., 2020; Gabriel et al., 2020).”
Survey of Hallucination in Natural Language Generation (2022, background): “[39] further analyze the FEQA and ind that the efectiveness of QA-based metrics depends on the question.”
Planning with Learned Entity Prompts for Abstractive Summarization (2021, background): “, 2020), plays a key role in aligning summary content to the input, yet undesired hallucinations are common in generated summaries (Maynez et al., 2020; Kryscinski et al., 2020; Gabriel et al., 2021).”
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization (2021, background): “Alongside improving factuality, we have also witnessed growing interests in automated factu-ality evaluation, since popular word-matching-based metrics, e.g., ROUGE, correlate poorly with human-rated factual consistency levels (Gabriel et al., 2021; Fabbri et al., 2021).”
AlignScore: Evaluating Factual Consistency with A Unified Alignment Function (2023, methodology): “Following Deng et al. (2021), Fabbri et al. (2022), Zhong et al. (2022) and Gabriel et al. (2021), we evaluate factual consistency metrics using TRUE benchmark (Honovich et al., 2022a) (consists of 11 datasets in diverse domains), SummaC benchmark (Laban et al., 2022) (includes 6 large…”
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization (2021, background): “(2020) propose a typology of errors found in summarization models, while Gabriel et al. (2021) propose a framework for meta-evaluation of faithfulness consistency metrics.”
DialSummEval: Revisiting Summarization Evaluation for Dialogues (2022, background): “Notably, Gabriel et al. (2021) is one of the few current studies using the dialogue summarization dataset SAMSum (Gliwa et al., 2019) for meta-evaluation, but it focuses on factual consistency and selects a small number of metrics.”
Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark (2021, background): “These results are in line with the robustness issues reported for other natural language generation metrics, despite the high correlation of those metrics with human judgments (Durmus et al., 2022; Gehrmann et al., 2021; Gabriel et al., 2021; Yeh et al., 2021).”
TRUE: Re-evaluating Factual Consistency Evaluation (2022, background): “Other works on meta-evaluation of factual consistency across datasets include GO-FIGURE (Gabriel et al., 2021) FRANK (Pagnoni et al.”
Spurious Correlations in Reference-Free Evaluation of Text Generation (2022, background): “Gabriel et al. (2021) has proposed a meta-evaluation framework to evaluate the evaluation metrics looking at certain aspects of these metrics such as robustness, sensitivity, high correlation with human scores, etc., and measured existing evaluation metrics across these aspects.”
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries (2021, background): “Thus, evaluating the factual consistency of the generated summaries with respect to the source is an important task (Falke et al., 2019; Cao et al., 2020; Gabriel et al., 2021; Durmus et al., 2020; Huang et al., 2021; Pagnoni et al., 2021).”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, background): “Other studies analyze a broader set of metrics (Peyrard, 2019; Bhandari et al., 2020; Deutsch and Roth, 2020; Fabbri et al., 2022a; Gabriel et al., 2021; Kasai et al., 2022b), including those specific to factual consistency evaluation (Kryscinski”
Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization (2021, background): “, 2020), there has been significant effort to develop automated metrics that can detect such errors (Zhou et al., 2021; Gabriel et al., 2021; Pagnoni et al., 2021a).”
Focus Attention: Promoting Faithfulness and Diversity in Summarization (2021, background): “For example, generating summaries that are faithful to the input is an unsolved problem (Kryscinski et al., 2020; Maynez et al., 2020; Gabriel et al., 2020).”