Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary

By Daniel Deutsch, Tania Bedrax-Weiss, Dan Roth. 2020

145 citations recorded by the source corpus.

Abstract A desirable property of a reference-based evaluation metric that measures the content quality of a summary is that it should estimate how much information that summary has in common with a reference. Traditional text overlap based metrics such as ROUGE fail to achieve this because they are limited to matching tokens, either lexically or via embeddings. In this work, we propose a metric to evaluate the content quality of a summary using question-answering (QA). QA-based methods directly measure a summary’s information overlap with a reference, making them fundamentally different than text overlap metrics. We demonstrate the experimental benefits of QA-based metrics through an analysis of our proposed metric, QAEval. QAEval outperforms current state-of-the-art metrics on most evaluations using benchmark datasets, while being competitive on others due to limitations of state-of-the-art models. Through a careful analysis of each component of QAEval, we identify its performance bottlenecks and estimate that its potential upper-bound performance surpasses all other automatic metrics, approaching that of the gold-standard Pyramid Method.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “However, this process can lead to artificially high correlation of the results with automatic metrics, which could overstate their effectiveness and the quality of human annotations (Deutsch et al., 2021b). Moreover, since NLG models may only differ in very few examples, statistical analyses should ”
Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics (2022, methodology): “10 and 11 contain the r SYS ∆( (cid:96), u ) correlations for ROUGE, BERTScore, and QAEval for various combinations of (cid:96) and u on both the Summ-Eval and REALSumm datasets.”
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization (2021, methodology): “This finding demonstrates the importance of determining question answerability, a point also emphasized in Deutsch et al. (2020) for QA-based metrics of relevance.”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, methodology): “We note that we use the permutation test instead of the paired bootstrapping test to calculate the statistical significance for metric comparison, since Deutsch et al. (2021b) found that permutation test works better for detecting significant results in metric comparison.”
Understanding the Extent to which Content Quality Metrics Measure the Information Quality of Summaries (2021, methodology): “…2008), BEwT-E (Tratz and Hovy, 2008), MeMoG (Giannakopou-los and Karkaletsis, 2011), METEOR (Denkowski and Lavie, 2014), MoverScore (Zhao et al., 2019), NPowER (Giannakopoulos and Karkaletsis, 2013), PyrEval (Gao et al., 2019), QAEval (Deutsch et al., 2021), ROUGE-2, and S 3 (Peyrard et al., 2017).”
TRUE: Re-evaluating Factual Consistency Evaluation (2022, methodology): “…and methods that target a speciﬁc task or domain, such as summarization (Falke et al., 2019; Kryscinski et al., 2020; Wang et al., 2020; Scialom et al., 2021; Deutsch et al., 2021; Xie et al., 2021) or dialogue (Dziri et al., 2021; Honovich et al., 2021; Nie et al., 2021; Qin et al., 2021).”
News Summarization and Evaluation in the Era of GPT-3 (2022, unknown): “We report both exact match (EM) and F1 components of QAEval.”
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023, methodology): “, 2019; Thompson and Post, 2020) and summarization (Bhandari et al., 2020; Deutsch et al., 2021).”
Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection (2021, unknown): “Recent studies explore categorical, content-based analysis for measuring the faithfulness of summaries (Goyal and Durrett, 2020; Deutsch and Roth, 2020).”
Finding a Balanced Degree of Automation for Summary Evaluation (2021, methodology): “However, Deutsch and Roth (2021) point out that the n-gram based metrics indicate more topic similarity than information overlap.”
Measuring the Feasibility of a Question and Answering System for the Sarawak Gazette Using Chatbot Technology (2025, unknown): “More recent evidence (Deutsch et al, 2021) highlights that ROUGE does not truly measure the amount of information that is common between two pieces of text, instead this metric simply measures the degree of overlap between them.”