Results of the WMT17 Metrics Shared Task

By Ondrej Bojar, Yvette Graham, Amir Kamran. 2017

141 citations recorded by the source corpus.

This paper presents the results of the WMT17 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in the WMT17 news translation task and Neural MT training task. We collected scores of 14 metrics from 8 research groups. In addition to that, we computed scores of 7 standard metrics (BLEU, SentBLEU, NIST, WER, PER, TER and CDER) as baselines. The collected scores were evaluated in terms of system-level correlation (how well each metric’s scores correlate with WMT17 official manual ranking of systems) and in terms of segment level correlation (how often a metric agrees with humans in judging the quality of a particular sentence). This year, we build upon two types of manual judgements: direct assessment (DA) and HUME manual semantic judgements.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “After promising results, Bojar et al. (2017) only used DA on a 0-100 scale and HUME.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “To address the issue, the machine translation community has been organizing yearly metrics shared tasks which produce metrics that achieve a high correlation (Stanojević et al., 2015; Bojar et al., 2016, 2017; Ma et al., 2018, 2019; Mathur et al., 2020b).”
BLEURT: Learning Robust Metrics for Text Generation (2020, methodology): “First, we benchmark B LEURT against existing text generation metrics on the last 3 years of the WMT Metrics Shared Task (Bojar et al., 2017).”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, background): “1 Note that, in comparison, the well established Conference in Machine Translation WMT’17 (running since 2006) got 31 institutions submitting to a total of 8 tasks (Bojar et al. 2017).”
Findings of the 2017 Conference on Machine Translation (WMT17) (2017, background): “…machine translation and multilin- gual image description (Elliott et al., 2017) • biomedical translation (Jimeno Yepes et al., 2017) • neural MT training (Bojar et al., 2017b) • bandit learning (Sokolov et al., 2017) In the news translation task (Section 2), participants were asked to…”
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2019, methodology): “Data We obtain the source language sentences, their system and reference translations from the WMT 2017 news translation shared task (Bojar et al., 2017).”
Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges (2019, methodology): “Starting from Bojar et al. (2017), when WMT fully switched to DA, we had to come up with a solid golden standard for segment-level judgements.”
Findings of the E2E NLG Challenge (2018, methodology): “Following previous shared tasks in related ﬁelds (Bojar et al., 2017b; Chen et al., 2015), we selected a range of metrics measuring word-overlap between system output and references, including BLEU, NIST, METEOR, ROUGE-L, and CIDEr.”
COMET: A Neural Framework for MT Evaluation (2020, background): “the number of annotations per segment is limited (Bojar et al., 2017b; Ma et al., 2018, 2019).”
RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation (2018, methodology): “…for training sentence embeddings. cs-en de-en ﬁ-en lv-en ro-en ru-en tr-en zh-en (Stanojevi´c et al., 2015), WMT16 (Bojar et al., 2016), and WMT17 (Bojar et al., 2017). cs-en de-en ﬁ-en lv-en ro-en ru-en tr-en zh-en WMT16 systems 6 10 9 - 7 10 8 - sentences 2,999 2,999 3,000 - 2,998 1,999 3,000…”
Results of the WMT20 Metrics Shared Task (2020, methodology): “Starting from Bojar et al. (2017), when WMT fully switched to DA, we had to come up with a solid golden standard for segment-level judgements.”
The First Multilingual Surface Realisation Shared Task (SR’18): Overview and Evaluation Results (2018, methodology): “For conformity with what has emerged as a new affordable human evaluation standard over the past two years in the main machine translation shared tasks held at WMT (Bojar et al., 2017a), we changed this design to look as follows, with the pointer starting at 0: Test data sets for human evaluations: ”
Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain (2021, methodology): “We use Kendall’s tau-like correlation, typically used for DA RR (Bojar et al., 2017; Ma et al., 2018, 2019; Mathur et al., 2020b), for evaluating the primary submissions on the challenge sets.”
A Survey of Evaluation Metrics Used for NLG Systems (2020, methodology): “Taking these limitations of Pearson and rank based correlations into consideration, the organizers of the WMT Metrics Shared Task [13, 98, 100] used a custom variant of Kendall’s τ to account for potential noise in the human ratings.”
On the use of BERT for Neural Machine Translation (2019, background): “ince it is corpus-level metric. Ideally, we would like to measure a number of sentences or a margin for which we observe an ‘important decrease’ in translation quality. According to Ma et al. (2018); Bojar et al. (2017), sentence level chrF achieves good correlation with human judgements for En-De n”
WMDO: Fluency-based Word Mover’s Distance for Machine Translation Evaluation (2019, methodology): “We used the WMT17 segment-level into-English datasets for our experiments (Bojar et al., 2017).”
Generative Pretraining for Paraphrase Evaluation (2021, background): “[19] Ondřej Bojar, Yvette Graham, and Amir Kamran.”