QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization

By Alexander R. Fabbri, C. Wu, Wenhao Liu, Caiming Xiong. 2021

313 citations recorded by the source corpus.

Factual consistency is an essential quality of text summarization models in practical settings. Existing work in evaluating this dimension can be broadly categorized into two lines of research, entailment-based and question answering (QA)-based metrics, and different experimental setups often lead to contrasting conclusions as to which paradigm performs the best. In this work, we conduct an extensive comparison of entailment and QA-based metrics, demonstrating that carefully choosing the components of a QA-based metric, especially question generation and answerability classification, is critical to performance. Building on those insights, we propose an optimized metric, which we call QAFactEval, that leads to a 14% average improvement over previous QA-based metrics on the SummaC factual consistency benchmark, and also outperforms the best-performing entailment-based metric. Moreover, we find that QA-based and entailment-based metrics can offer complementary signals and be combined into a single metric for a further performance boost.

Subfields: Artificial Intelligence, General Social Sciences, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Many works like that of Fabbri et al. (2021) would not be possible without access to model outputs, and such corpora can be used for metric development and validation, and to conduct meta evaluations.”
Large Language Model Alignment: A Survey (2023, background): “While this could be done on a single task (Laban et al., 2022; Fabbri et al., 2021), consistency evaluation on multiple tasks is more convincing.”
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023, methodology): “This includes model-based evaluation in summarization that considers the consistency between a summary and a source document using QA or NLI (Kryscin-ski et al., 2020; Wang et al., 2020; Fabbri et al., 2022; Deutsch et al., 2021; Laban et al., 2022).”
News Summarization and Evaluation in the Era of GPT-3 (2022, methodology): “QA-based metrics (Fabbri et al., 2022; Durmus et al., 2020) and trained factuality models (Kryscinski et al.”
ExpertQA: Expert-Curated Questions and Attributed Answers (2023, background): “We establish several baselines and show that we can improve models by finetuning on E XPERT QA but that there is substantial room for improvement, both in terms of ROUGE and QAFactEval (§6).”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, background): “…Fabbri et al., 2022a; Gabriel et al., 2021; Kasai et al., 2022b), including those specific to factual consistency evaluation (Kryscinski et al., 2020; Durmus et al., 2020; Wang et al., 2020; Maynez et al., 2020; Laban et al., 20d; Fabbri et al., 2022b; Honovich et al., 2022; Tam et al., 2022).”
FacTool: Factuality Detection in Generative AI - A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios (2023, background): “, 2020), hallucination detection models for text summarization (Fabbri et al., 2022), and execution-based evaluation for code (Shi et al.”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, unknown): “Building on this foundational work, Fabbri et al. (2022) conducted an in-depth evaluation of the components within QA-based metrics, yielding further enhancements in faithfulness evaluation.”
Benchmarking Large Language Models for News Summarization (2023, methodology): “For CNN/DM, we observe that the reference-based automatic metrics have a moderate correlation with some aspects of human judgments, e.g., Rouge-L has a 0.72 Kendall’s tau correlation co-efﬁcient with relevance in Table 3.”
TRUE: Re-evaluating Factual Consistency Evaluation (2022, result): “Summarization - FRANK (Pagnoni et al., 2021) 671 + 33.2% - SummEval (Fabbri et al., 2021) 1,600 - 81.6% - MNBM (Maynez et al., 2020) 2,500 - 10.2% - QAGS-CNNDM (Wang et al., 2020) 235 - 48.1% - QAGS-XSum (Wang et al., 2020) 239 - 48.5% Dialogue - BEGIN (Dziri et al., 2021) 836 + 33.7% -”
AlignScore: Evaluating Factual Consistency with A Unified Alignment Function (2023, methodology): “The results of G-EVAL, GPTScore and ChatGPT are from Liu et al. (2023), Fu et al. (2023), and Gao et al. (2023). forms previous metrics in terms of overall performance, including the competitive QAFactEval and UniEval metrics, dominating 6 out of 7 datasets.”
ASTRA-QA: A Benchmark for Abstract Question Answering over Documents (2026, unknown): “Early evaluation of answers to abstract questions relied on lexical or semantic reference matching, such as ROUGE [17], BERTScore [18], and QA-based or factuality metrics such as FEQA [27], QuestEval [19], QAFactEval [20], and FActScore [21].”