Understanding the Extent to which Content Quality Metrics Measure the Information Quality of Summaries

By Daniel Deutsch, Dan Roth. 2021

33 citations recorded by the source corpus.

Reference-based metrics such as ROUGE or BERTScore evaluate the content quality of a summary by comparing the summary to a reference. Ideally, this comparison should measure the summary’s information quality by calculating how much information the summaries have in common. In this work, we analyze the token alignments used by ROUGE and BERTScore to compare summaries and argue that their scores largely cannot be interpreted as measuring information overlap. Rather, they are better estimates of the extent to which the summaries discuss the same topics. Further, we provide evidence that this result holds true for many other summarization evaluation metrics. The consequence of this result is that the most frequently used summarization evaluation metrics do not align with the community’s research goal, to generate summaries with high-quality information. However, we conclude by demonstrating that a recently proposed metric, QAEval, which scores summaries using question-answering, appears to better capture information quality than current evaluations, highlighting a direction for future research.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Similar points are made by Deutsch and Roth (2021) who show that many similarity metrics capture the overlap in topics between two summaries much better than the overlap in their information.”
Benchmarking Large Language Models for News Summarization (2023, background): “Many automatic metrics have been proposed (Lin, 2004; Zhang* et al., 2020; Sellam et al., 2020; Durmus et al., 2020; Maynez et al., 2020; Deutsch and Roth, 2021) but they do not always correlate with human evaluation of summarization systems (Fab-bri et al., 2020; Durmus et al., 2022).”
News Summarization and Evaluation in the Era of GPT-3 (2022, result): “Although issues with these metrics, particularly low correlation with human judgments, have also been studied earlier (Fabbri et al., 2021; Deutsch and Roth, 2021), they are considered reliable when comparing systems in different score ranges (Peyrard, 2019; Deutsch et al.”
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization (2021, background): “Recent work has shown that QA-based metrics better measure the overlap of information units for determining summary relevance over embedding-based metrics (Deutsch and Roth, 2021), further driving our study of QA-based metrics for factual consistency.”