BLEU Might Be Guilty but References Are Not Innocent

By Markus Freitag, David Grangier, Isaac Caswell. 2020

169 citations recorded by the source corpus.

The quality of automatic metrics for machine translation has been increasingly called into question, especially for high-quality systems. This paper demonstrates that, while choice of metric is important, the nature of the references is also critical. We study different methods to collect references and compare their value in automated evaluation by reporting correlation with human evaluation for a variety of systems and metrics. Motivated by the finding that typical references exhibit poor diversity, concentrating around translationese language, we develop a paraphrasing task for linguists to perform on existing reference translations, which counteracts this bias. Our method yields higher correlation with human judgment not only for the submissions of WMT 2019 English to German, but also for Back-translation and APE augmented MT output, which have been shown to have low correlation with automatic metrics using standard references. We demonstrate that our methodology improves correlation with all modern evaluation metrics we look at, including embedding-based methods. To complete this picture, we reveal that multi-reference BLEU does not improve the correlation for high quality output, and present an alternative multi-reference formulation that is more effective.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Similarly, Freitag et al. (2020) show that BLEU, METEOR, and BERTSCORE may fail to reward good translations when the reference text contains artifacts such as “translationese”.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “Moreover, the reliability of these metrics depends on the quality and number of the references (Mathur et al., 2020a; Freitag et al., 2020).”
Scaling Laws for Neural Machine Translation (2021, background): “Earlier work (Freitag et al., 2019; 2020; Graham et al., 2020) showed that it is important to differentiate between the two different kinds of test sets as the style of natural sentences and human (or machine) translations (translationese) is quite different.”
Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation (2021, methodology): “Human-P is a reference translation generated using the paraphrasing method of (Freitag et al., 2020) which asked linguists to paraphrase existing reference translations as much as possible while also suggesting using synonyms and different sentence structures.”
XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning (2020, background): “the structural and lexical properties of the source language: the resulting evaluation benchmarks thus fail to measure system performance adequately (Koppel and Ordan, 2011; Volansky et al., 2015; Artetxe et al., 2020a; Freitag et al., 2020).”
Results of the WMT20 Metrics Shared Task (2020, methodology): “To overcome this problem, Freitag et al. (2020) introduced paraphrased references that also value the translation quality of human translations and alternative (less simple/monotonic) MT output.”
Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking (2021, background): “This has led to an outpouring of new benchmarks designed not only to evaluate models on new tasks, but also to address weaknesses in existing models [49, 55, 34], and expose artifacts in existing benchmarks [42, 19, 25, 33, 22, 37].”
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (2024, background): “However, these conventional metrics are prone to false negatives because they are not expressive enough to recognize responses that are of good quality but differ from the reference answer (Schluter, 2017; Freitag et al., 2020; Hanna and Bojar, 2021).”
Benchmarking Large Language Models for News Summarization (2023, background): “This points to the important role of quality reference summaries for reference-based metrics, as previously observed in machine translation (Freitag et al., 2020).”
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand (2021, background): “These para-phrased translations are shown to increase corre-6 lations with human judgments by mitigating the translationese effect and diversifying the reference when the generation quality is measured by reference-based metrics (Freitag et al., 2020).”
Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task (2020, methodology): “The model was trained for 880K steps instead of 1 million, and it uses a sequence length of 256 tokens instead of 128. with the standard WMT references as well as the paraphrased reference from Freitag et al. (2020).”
Control Prefixes for Parameter-Efficient Text Generation (2021, background): “For example, Freitag et al. (2020) and Mathur et al. (2020) find that when comparing two high-quality systems, differences according to a metric may also reflect how the references are written or flaws in the metric itself.”
How well do you know your summarization datasets? (2021, unknown): “While such metrics exhibit high correlation with human-judgement, a low score does not necessarily imply an incorrect generation, as demonstrated by Freitag et al. (2020) for machine translation.”
Translationese as a Language in “Multilingual” NMT (2019, background): “The effects of translationese on MT training and evaluation have been investigated by many prior authors (Kurokawa et al., 2009; Lembersky et al., 2012; Toral et al., 2018; Zhang and Toral, 2019; Graham et al., 2019; Freitag et al., 2019; Edunov et al., 2019; Freitag et al., 2020).”
Building Machine Translation Systems for the Next Thousand Languages (2022, background): “…find, outputs of unsupervised NMT are less monotonic and more natural than outputs from supervised NMT, which, like the findings on paraphrased references from Freitag et al. (2020), produce BLEU scores that are much lower — although sometimes better correlated with human judgements of quality.”
The statistical advantage of automatic NLG metrics at the system level (2021, background): “A growing body of MT literature focuses on professional linguists (Freitag et al., 2020; Mathur et al., 2020b), and there are at least two known properties of their judgments: their judgments have better interannotator agreement (contain less noise), and they are more sensitive to linguistic…”
STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation (2020, background): “Recently, the discriminative power of BLEU has been called into question when evaluating state-of-the-art machine translation systems, leading researchers to investigate alternative evaluation metrics (Freitag et al., 2020; Sellam et al., 2020).”