Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics

By Nitika Mathur, Tim Baldwin, Trevor Cohn. 2020

308 citations recorded by the source corpus.

Automatic metrics are fundamental for the development and evaluation of machine translation systems. Judging whether, and to what extent, automatic metrics concur with the gold standard of human evaluation is not a straightforward problem. We show that current methods for judging metrics are highly sensitive to the translations used for assessment, particularly the presence of outliers, which often leads to falsely confident conclusions about a metric’s efficacy. Finally, we turn to pairwise system ranking, developing a method for thresholding performance improvement under an automatic metric against human judgements, which allows quantification of type I versus type II errors incurred, i.e., insignificant human differences in system quality that are accepted, and significant human differences that are rejected. Together, these findings suggest improvements to the protocols for metric evaluation and system performance evaluation in machine translation.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Learning Compact Metrics for MT (2021, neutral): “E XTENDED , three SOTA metrics from WMT Metrics ’20 (Mathur et al., 2020b).”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “Moreover, the reliability of these metrics depends on the quality and number of the references (Mathur et al., 2020a; Freitag et al., 2020).”
What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers (2021, neutral): “As shown in Table 8, HyperCLOVA can yield high-quality titles comparable to GT. Interestingly, we ﬁnd that higher BLEU scores with respect to GT do not guarantee higher qualities (Mathur et al., 2020).”
Evaluation of Text Generation: A Survey (2020, background): “Recent work by (Mathur et al., 2020) investigated how sensitive the machine translation evaluation metrics are to outliers.”
BARTScore: Evaluating Generated Text as Text Generation (2021, background): “As stated above, our goal is to assess the quality of generated text [2, 39].”
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing (2020, background): “…automatic metrics helped enable are now causing the correlation between human judgments and automatic metrics to break down (Ma et al., 2019; Mathur et al., 2020) especially for BLEU (Papineni et al., 2002), which has been the de facto standard metric since its introduction almost two…”
Results of the WMT20 Metrics Shared Task (2020, methodology): “To remove outliers, we are guided by the robust outlier detection method proposed for MT metric evaluation by Mathur et al. (2020).”
Control Prefixes for Parameter-Efficient Text Generation (2021, background): “For example, Freitag et al. (2020) and Mathur et al. (2020) find that when comparing two high-quality systems, differences according to a metric may also reflect how the references are written or flaws in the metric itself.”
Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR (2020, background): “However, such metrics are suffer from several issues, for example, they are highly sensitive to the translations used for assessment which may easily lead to falsely conﬁdent conclusions about a metrics efﬁcacy (Callison-Burch et al., 2006; Mathur et al., 2020).”
Unbabel’s Participation in the WMT20 Metrics Shared Task (2020, methodology): “Our Estimators generally follow the architecture proposed in Rei et al. (2020), that is to say we encode segment-level representations using XLM-RoBERTa and pass these outputs through a feed-forward regressor.”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “While, for those text generation tasks involving contextual understanding and reasoning (e.g., story generation), BLEU neglects semantic meaning and sentence structure and shows limitations in morphologically complex languages [162].”
Reward Optimization for Neural Machine Translation with Learned Metrics (2021, background): “Mathur et al. (2020) reports that the model-based metrics outperform BLEU in almost all situations.”
Evaluating the Morphosyntactic Well-formedness of Generated Texts (2021, background): “We follow Mathur et al. (2020) to remove outlier systems, since they tend to significantly boost the correlation scores, making the correlations unreliable, especially for the best performing systems (Ma et al., 2019).”
Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain (2021, methodology): “We use Kendall’s tau-like correlation, typically used for DA RR (Bojar et al., 2017; Ma et al., 2018, 2019; Mathur et al., 2020b), for evaluating the primary submissions on the challenge sets.”
Machine Translation into Low-resource Language Varieties (2021, methodology): “While we recognize the limitations of BLEU (Mathur et al., 2020), more sophisticated embedding-based metrics for MT evaluation (Zhang et al., 2020; Sellam et al., 2020) are simply not available for language varieties.”
The statistical advantage of automatic NLG metrics at the system level (2021, background): “Recently, Mathur et al. (2020a) highlights two issues with the use of correlation: First, Pearson’s r is neither interpretable nor reﬂective of system-level metric use in practice.”
Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task (2020, background): “…the use of n-gram overlap metrics such as BLEU, which focus solely on surface-level aspects of the generated text, and thus may correlate poorly with human evaluation (Papineni et al., 2002; Lin, 2004; Ma et al., 2019; Mathur et al., 2020; Belz and Reiter, 2006; Callison-Burch et al., 2006).”
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand (2021, methodology): “Moreover, if a metric can perform well on an instance level, it can be used to augment human judgments by, for example, ﬂagging likely wrong ratings (Mathur et al., 2020b).”