Spurious Correlations in Reference-Free Evaluation of Text Generation

By Esin Durmus, Faisal Ladhak, Tatsunori B. Hashimoto. 2022

40 citations recorded by the source corpus.

Model-based, reference-free evaluation metricshave been proposed as a fast and cost-effectiveapproach to evaluate Natural Language Generation(NLG) systems. Despite promising recentresults, we find evidence that reference-freeevaluation metrics of summarization and dialoggeneration may be relying on spuriouscorrelations with measures such as word overlap,perplexity, and length. We further observethat for text summarization, these metrics havehigh error rates when ranking current state-ofthe-art abstractive summarization systems. Wedemonstrate that these errors can be mitigatedby explicitly designing evaluation metrics toavoid spurious features in reference-free evaluation.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Dušek et al. (2019) find that cleaning the E2E NLG dataset (Novikova et al.”
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (2024, background): “While this latter class of algorithms has become sufficiently accurate that they match inter-annotator agreement rates, other works have shown that such measurements are heavily confounded by spurious correlations such as perplexity and length (Durmus et al., 2022).”
News Summarization and Evaluation in the Era of GPT-3 (2022, background): “Such shortcomings of reference-free evaluation metrics due to spurious correlations have also been studied in prior work (Durmus et al., 2022).”
Benchmarking Large Language Models for News Summarization (2023, methodology): “In this work, we evaluate the effectiveness of automatic metrics for evaluating LLMs and show that the usefulness of reference-based evaluation is closely linked to the quality of the references.”
Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark (2021, background): “These results are in line with the robustness issues reported for other natural language generation metrics, despite the high correlation of those metrics with human judgments (Durmus et al., 2022; Gehrmann et al., 2021; Gabriel et al., 2021; Yeh et al., 2021).”