QuestEval: Summarization Asks for Fact-based Evaluation

By Thomas Scialom, Paul-Alexis Dray, Patrick Gallinari, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang. 2021

371 citations recorded by the source corpus.

Summarization evaluation remains an open research problem: current metrics such as ROUGE are known to be limited and to correlate poorly with human judgments. To alleviate this issue, recent work has proposed evaluation metrics which rely on question answering models to assess whether a summary contains all the relevant information in its source document. Though promising, the proposed approaches have so far failed to correlate better than ROUGE with human judgments. In this paper, we extend previous approaches and propose a unified framework, named QuestEval. In contrast to established metrics such as ROUGE or BERTScore, QuestEval does not require any ground-truth reference. Nonetheless, QuestEval substantially improves the correlation with human judgments over four evaluation dimensions (consistency, coherence, fluency, and relevance), as shown in extensive experiments.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…fixed size n-grams, but other basic element metrics like PARENT (Dhingra et al., 2019) only focus on content words. generate questions (Chen et al., 2018; Wang et al., 2020; Durmus et al., 2020; Scialom et al., 2021; Rebuffel et al., 2021; Honovich et al., 2021; Deutsch et al., 2021a, inter alia).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “, 2020; Goyal and Durrett, 2020), or question-answering errors using targeted QA models (Scialom et al., 2021).”
PEER: A Collaborative Language Model (2022, methodology): “To evaluate the faithfulness of all models to the provided documents, we also consider QuestEval (QE) scores (Scialom et al., 2021),”
Survey of Hallucination in Natural Language Generation (2022, methodology): “It is already put in use to evaluate hallucinations in many tasks, such as summarization [36, 164, 191], dialogue [73], and data2text generation [155].”
GPTScore: Evaluate as You Desire (2023, background): “…refers to one evaluator handle several evaluation aspects by using different input and output text pair (Yuan et al., 2021), different prompt designed by the aspect name (Zhong et al., 2022; Mehri and Eskénazi, 2020a), (Mehri and Eskénazi, 2020b), different formulas (Scialom et al., 2021).”
A Comprehensive Assessment of Dialog Evaluation Metrics (2021, methodology): “…automatic metrics speciﬁcally designed for dialog evaluation, this paper also evaluates the performance of BERTScore (Zhang et al., 2019), QuestEval (Scialom et al., 2021), and BLEURT (Sellam et al., 2020) which were originally designed for evaluating machine translation, summarization and general…”
News Summarization and Evaluation in the Era of GPT-3 (2022, methodology): “We report the performance of summarization systems using two QA-based metrics: QuestEval (Scialom et al., 2021) and QAFactEval (Fabbri et al., 2022).”
AlignScore: Evaluating Factual Consistency with A Unified Alignment Function (2023, methodology): “We include the latest QAFactEval (Fabbri et al., 2022), QuestEval (Scialom et al., 2021), and FEQA (Durmus et al., 2020) as our baselines.”
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization (2021, methodology): “QuestEval (Scialom et al., 2021) is a QAG method taking both precision and recall into account.”
CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning (2021, background): “, 2021): Because ROUGE scores only measure token overlap, other automated metrics (Rebuffel et al., 2021; Kryscinski et al., 2020; Wang et al., 2020; Scialom et al., 2021) have been proposed to evaluate faithfulness more precisely.”
Factual Consistency Evaluation for Text Summarization via Counterfactual Estimation (2021, methodology): “For QA-based metrics QAGS and QuestEval, following the settings in the original papers, the QG model and QA model in QAGS are implemented based on BART and BERT respectively, and for QuestEval, they are both implemented based on T5 (Raffel et al., 2020).”
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way (2022, background): “This mismatch may contribute to the poor correlation of ROUGE with human judgments of quality for these datasets (Pang et al., 2021a; Pagnoni et al., 2021; Scialom et al., 2021, i.a.).”
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization (2021, background): “While several QA-based metrics with similar structures have been applied for factual consistency, (Durmus et al., 2020; Wang et al., 2020b; Scialom et al., 2021), they differ in their underlying answer selection, question generation, question answering, and answer overlap components, reporting…”
Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022, background): “Uniﬁed Evaluator Several recent evaluators can predict multiple numbers for evaluating text by using different input and output contents (Yuan et al., 2021), multiple model variants (Mehri and Eskenazi, 2020), or different formulas (Scialom et al., 2021), and we refer to them as uniﬁed evaluators.”
TRUE: Re-evaluating Factual Consistency Evaluation (2022, methodology): “…datasets and methods that target a speciﬁc task or domain, such as summarization (Falke et al., 2019; Kryscinski et al., 2020; Wang et al., 2020; Scialom et al., 2021; Deutsch et al., 2021; Xie et al., 2021) or dialogue (Dziri et al., 2021; Honovich et al., 2021; Nie et al., 2021; Qin et al.,…”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, neutral): “Notable implementations include (Durmus et al., 2020; Wang et al., 2020a; Scialom et al., 2021; Honovich et al., 2021).”
Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation (2021, methodology): “In its recent extension, Q UEST E VAL (Scialom et al., 2021) was shown to overperform standard metrics in Text-vs-Text tasks such as the evaluation of summaries.”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, methodology): “We plan to report QuestEval (Scialom et al., 2021) in a future version.”