Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust

By Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu (羅致翹) Lo, Craig Alan Stewart, Eleftherios Avramidis, Tom Kocmi, George F. Foster, Alon Lavie, André F. T. Martins. 2022

351 citations recorded by the source corpus.

This paper presents the results of the WMT22 Metrics Shared Task. Participants submitting automatic MT evaluation metrics were asked to score the outputs of the translation systems competing in the WMT22 News Translation Task on four different domains: news, social, ecommerce, and chat. All metrics were evaluated on how well they correlate with human ratings at the system and segment level.Similar to last year, we acquired our own human ratings based on expert-based human evaluation via Multidimensional Quality Metrics (MQM). This setup had several advantages, among other things: (i) expert-based evaluation is more reliable, (ii) we extended the pool of translations by 5 additional translations based on MBR decoding or rescoring which are challenging for current metrics. In addition, we initiated a challenge set subtask, where participants had to create contrastive test suites for evaluating metrics’ ability to capture and penalise specific types of translation errors.Finally, we present an extensive analysis on how well metrics perform on three language pairs: English to German, English to Russian and Chinese to English. The results demonstrate the superiority of neural-based learned metrics and demonstrate again that overlap metrics like Bleu, spBleu or chrf correlate poorly with human ratings. The results also reveal that neural-based metrics are remarkably robust across different domains and challenges.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

PaLM 2 Technical Report (2023, background): “, 2002) due to BLEU’s poor correlation with human judgements of quality, especially for high-quality translations (Freitag et al., 2022).”
Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models (2023, methodology): “In the MQM2022 test set (English-German, English-Russian, and Chinese-English) [14], a scoring task was performed with a classiﬁcation task to compare the accuracy [39] and kendall tau scores [15] of seven GPT models under four cue templates.”
Reinforced Self-Training (ReST) for Language Modeling (2023, methodology): “We considered two types of reward models for translation (Freitag et al., 2022): i) Reference-free reward models estimate of how good a translation is based on the source and candidate sentences only, ii) Reference-based reward models additionally use the reference human translation to decide how…”
Large Language Models Are State-of-the-Art Evaluators of Translation Quality (2023, methodology): “To measure the performance of the proposed GEMBA metric, we follow the methodology and use test data provided by the WMT22 Metrics shared task (Freitag et al., 2022b) which hosts an annual evaluation of automatic metrics, benchmarking them against human gold labels.”
Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models (2023, methodology): “In the MQM2022 test set (English-German, English-Russian, and Chinese-English) [15], a scoring task was performed with a classification task to compare the accuracy [42] and kendall tau scores [16] of seven GPT models under four cue templates.”
xcomet: Transparent Machine Translation Evaluation through Fine-grained Error Detection (2023, methodology): “These spans complement the sentence-level score by providing a detailed view into the translation errors. outputs are consistently among the best metrics for MT evaluation according to correlations with human judgments (Freitag et al., 2021b, 2022).”
COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task (2022, methodology): “Our efforts this year built on findings and observations from our participation in the WMT 2021 Metrics Shared Task (Rei et al., 2021; Freitag et al., 2021b) to further improve C OMET for the Metrics task and to increase its robustness to translation errors such as deviation from named entities, rev”