Metrics also Disagree in the Low Scoring Range: Revisiting Summarization Evaluation Metrics

By Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu. 2020

20 citations recorded by the source corpus.

In text summarization, evaluating the efficacy of automatic metrics without human judgments has become recently popular. One exemplar work (Peyrard, 2019) concludes that automatic metrics strongly disagree when ranking high-scoring summaries. In this paper, we revisit their experiments and find that their observations stem from the fact that metrics disagree in ranking summaries from any narrow scoring range. We hypothesize that this may be because summaries are similar to each other in a narrow scoring range and are thus, difficult to rank. Apart from the width of the scoring range of summaries, we analyze three other properties that impact inter-metric agreement - Ease of Summarization, Abstractiveness, and Coverage.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Bhandari et al. (2020a) revisits this conclusion, demonstrating that metrics disagree whenever the quality range is narrow, regardless of whether the summaries are good or bad. Building on this work, Deutsch et al. (2022) demonstrate that ROUGE and other recent metrics have a near-zero correlation w”
QuestEval: Summarization Asks for Fact-based Evaluation (2021, background): “For these reasons, n -gram based metrics, such as ROUGE (Lin, 2004), are known to poorly reﬂect human preference (Louis and Nenkova, 2013; Novikova et al., 2017; Paulus et al., 2017; Bhandari et al., 2020).”
BARTScore: Evaluating Generated Text as Text Generation (2021, background): “As stated above, our goal is to assess the quality of generated text [2, 39].”
SAFEval: Summarization Asks for Fact-based Evaluation (2021, background): “For these reasons, n -gram based metrics, such as ROUGE (Lin, 2004), are known to poorly reﬂect human preference (Louis and Nenkova, 2013; Novikova et al., 2017; Paulus et al., 2017; Bhandari et al., 2020).”
News Summarization and Evaluation in the Era of GPT-3 (2022, background): “be correlated with human preferences (Bhandari et al., 2020; Deutsch et al., 2022).”