Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics

By Daniel Deutsch, Rotem Dror, Dan Roth. 2022

53 citations recorded by the source corpus.

How reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system-level correlations. We identify two ways in which the definition of the system-level correlation is inconsistent with how metrics are used to evaluate systems in practice and propose changes to rectify this disconnect. First, we calculate the system score for an automatic metric using the full test set instead of the subset of summaries judged by humans, which is currently standard practice. We demonstrate how this small change leads to more precise estimates of system-level correlations. Second, we propose to calculate correlations only on pairs of systems that are separated by small differences in automatic scores which are commonly observed in practice. This allows us to demonstrate that our best estimate of the correlation of ROUGE to human judgments is near 0 in realistic scenarios. The results from the analyses point to the need to collect more high-quality human judgments and to improve automatic metrics when differences in system scores are small.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Summarization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Caiming Xiong
2. Pengfei Liu
3. Greg Durrett
4. Dragomir R. Radev
5. Tanya Goyal
6. Junyi Jessy Li
7. Graham Neubig
8. Mohit Bansal

In Evaluation & Benchmarks

1. Greg Durrett
2. Tanya Goyal
3. Junyi Jessy Li
4. Alexander R. Fabbri
5. Caiming Xiong
6. Pengfei Liu
7. Dragomir R. Radev
8. Yixin Liu

In Summarization

1. Yixin Liu
2. Pengfei Liu
3. Dragomir R. Radev
4. Greg Durrett
5. Tanya Goyal
6. Graham Neubig
7. Peng Liu

Citations in the corpus

• News Summarization and Evaluation in the Era of GPT-3 (2022, background): “be correlated with human preferences (Bhandari et al., 2020; Deutsch et al., 2022).”
• Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, methodology): “However, this benchmark only covers a single dataset (CNNDM) without a focus on similarly-performing state-of-the-art systems, which may skew metric analysis (Tang et al., 2022a) and not fully reflect realistic scenar- ios (Deutsch et al., 2022b).”