To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation

By Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, Arul Menezes. 2021

250 citations recorded by the source corpus.

Automatic metrics are commonly used as the exclusive tool for declaring the superiority of one machine translation system’s quality over another. The community choice of automatic metric guides research directions and industrial developments by deciding which models are deemed better. Evaluating metrics correlations with sets of human judgements has been limited by the size of these sets. In this paper, we corroborate how reliable metrics are in contrast to human judgements on – to the best of our knowledge – the largest collection of judgements reported in the literature. Arguably, pairwise rankings of two systems are the most common evaluation tasks in research or deployment scenarios. Taking human judgement as a gold standard, we investigate which metrics have the highest accuracy in predicting translation quality rankings for such system pairs. Furthermore, we evaluate the performance of various metrics across different language pairs and domains. Lastly, we show that the sole use of BLEU impeded the development of improved models leading to bad deployment decisions. We release the collection of 2.3M sentence-level human judgements for 4380 systems for further analysis and replication of our work.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, neutral): “More recently, Kocmi et al. (2021) assess how well automatic metrics compute pairwise rankings for MT systems, and recommend using a combination of overlap-based and pretraining-based metrics, confirming the previous findings that metrics may be used to rank MT models at the system-level.”
Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models (2023, methodology): “In the MQM2022 test set (English-German, English-Russian, and Chinese-English) [69], a scoring task was performed with a classification task to compare the accuracy [70] and kendall tau scores [71] of seven GPT models under four cue templates.”
Spurious Correlations in Reference-Free Evaluation of Text Generation (2022, methodology): “System-level evaluation: An alternative approach to evaluation is systems-level rankings (Mathur et al., 2020; Kocmi et al., 2021), which we deﬁne as the ability to identify which model is better amongst a set of models M .”
Findings of the 2021 Conference on Machine Translation (WMT21) (2021, background): “Recently, automatic metrics based on contextual embeddings have been shown to correlate better than string matching ones with human judgments (Kocmi et al., 2021).”
Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain (2021, methodology): “• Accuracy for ranking system pairs We calculate a joint score across all language pairs and adopt the pairwise accuracy score for ranking system pairs to generate the ﬁnal metric ranking (Kocmi et al., 2021).”
Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust (2022, methodology): “We computed three correlations: system-level pair-wise ranking accuracy (as proposed by Kocmi et al., 2021), Pearson and Kendall .”
COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task (2022, methodology): “To evaluate how our metrics perform we used the pairwise accuracy proposed in (Kocmi et al., 2021), which simulates a real world scenario where we are interested in comparing two systems and deciding which one is better.”
Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models (2023, methodology): “In the MQM2022 test set (English-German, English-Russian, and Chinese-English) [15], a scoring task was performed with a classification task to compare the accuracy [42] and kendall tau scores [16] of seven GPT models under four cue templates.”
Large Language Models Are State-of-the-Art Evaluators of Translation Quality (2023, methodology): “To determine how well automatic metrics correlate with humans, we measure system-level, pair-wise accuracy ( accuracy , Kocmi et al., 2021).”
SMART: Sentences as Basic Units for Text Evaluation (2022, background): “Unlike ROUGE and BLEU which operate at the token level, CHRF is more effective especially in morphologically-rich languages as it does not require any tokenization, lemmatization, and stemming (Mathur et al., 2020; Kocmi et al., 2021).”
Prompting PaLM for Translation: Assessing Strategies and Performance (2022, methodology): “…systems; (ii) we use a SOTA automatic metric ( BLEURT ) instead of BLEU which has been demonstrated to be suboptimal for high-quality translations (Kocmi et al., 2021; Freitag et al., 2021b); and (iii) we conduct an expert-based human evaluation with detailed categories to characterize the error…”