A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods

By Daniel Deutsch, Rotem Dror, Dan Roth. 2021

96 citations recorded by the source corpus.

Abstract The quality of a summarization evaluation metric is quantified by calculating the correlation between its scores and human annotations across a large number of summaries. Currently, it is unclear how precise these correlation estimates are, nor whether differences between two metrics’ correlations reflect a true difference or if it is due to mere chance. In this work, we address these two problems by proposing methods for calculating confidence intervals and running hypothesis tests for correlations using two resampling methods, bootstrapping and permutation. After evaluating which of the proposed methods is most appropriate for summarization through two simulation experiments, we analyze the results of applying these methods to several different automatic evaluation metrics across three sets of human annotations. We find that the confidence intervals are rather wide, demonstrating high uncertainty in the reliability of automatic metrics. Further, although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do so in some evaluation settings.1

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…fixed size n-grams, but other basic element metrics like PARENT (Dhingra et al., 2019) only focus on content words. generate questions (Chen et al., 2018; Wang et al., 2020; Durmus et al., 2020; Scialom et al., 2021; Rebuffel et al., 2021; Honovich et al., 2021; Deutsch et al., 2021a, inter alia).”
Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics (2022, methodology): “Although the above deﬁnition of the system-level correlation has been used by recent meta-evaluation studies of metrics (Bhandari et al., 2020; Fabbri et al., 2021; Deutsch et al., 2021b), there is a disconnect between how the automatic metrics are evaluated and how they are used in practice.”
Does Summary Evaluation Survive Translation to Other Languages? (2021, methodology): “We provide an illustrative example in Appendix C.) While permutation-based tests have been shown to have higher power in summarization evaluation than bootstrap resampling (Deutsch et al., 2021), permutation tests assume null hypothesis H 0 : ρ 1 = ρ 2 and are not simply adapted to our case.”
xcomet: Transparent Machine Translation Evaluation through Fine-grained Error Detection (2023, methodology): “At the sentence-level, we report Kendall’s Tau ( τ ) using the Perm-Both hypothesis test (Deutsch et al., 2021).”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, methodology): “We note that we use the permutation test instead of the paired bootstrapping test to calculate the statistical significance for metric comparison, since Deutsch et al. (2021b) found that permutation test works better for detecting significant results in metric comparison.”
Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust (2022, methodology): “For each task, we compare all pairs of metrics, and determine whether the difference in their correlation scores is significant according to the PERM-55 BOTH hypothesis test of Deutsch et al. (2021), using 1000 re-sampling runs, and setting p = 0 .”
Prompting PaLM for Translation: Assessing Strategies and Performance (2022, methodology): “All MQM results labelled with † are significantly better than all other systems based on PERM-BOTH pair-wise significance testing (Deutsch et al., 2021) with p = 0.”
Automatic Text Evaluation through the Lens of Wasserstein Barycenters (2021, methodology): “To ensure that observed improvement is statistically significant we follow common consensus (Deutsch et al., 2021; Graham, 2015; Graham et al., 2015; Graham and Baldwin, 2014) in NMT and rely on William test (Steiger, 1980) as considered observations are correlated3.”
Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain (2021, methodology): “We run PERM-BOTH hypothesis test (Deutsch et al., 2021) on the pairwise system-level accuracy of Table 8 to measure signiﬁcance between metrics’ performance.”
Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary (2020, background): “Those underlined are statistically indistinguishable from the highest under a single-tailed permutation test for correlations with α = 0.05 (Deutsch et al., 2021).”