Results of the WMT16 Metrics Shared Task

By Ondrej Bojar, Yvette Graham, Amir Kamran, Miloš Stanojević. 2016

127 citations recorded by the source corpus.

This paper presents the results of the WMT16 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in the WMT16 Shared Translation Task. We collected scores of 16 metrics from 9 research groups. In addition to that, we computed scores of 9 standard metrics (BLEU, SentBLEU, NIST, WER, PER, TER and CDER) as baselines. The collected scores were evaluated in terms of system-level correlation (how well each metric’s scores correlate with WMT16 official manual ranking of systems) and in terms of segment level correlation (how often a metric agrees with humans in comparing two translations of a particular sentence). This year there are several additions to the setup: large number of language pairs (18 in total), datasets from different domains (news, IT and medical), and different kinds of judgments: relative ranking (RR), direct assessment (DA) and HUME manual semantic judgments. Finally, generation of large number of hybrid systems was trialed for provision of more conclusive system-level metric rankings.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2016 Conference on Machine Translation (2016, methodology): “The IT-domain translation task introduced this year brought several novelties to WMT: • 4 out of the 7 languages of the IT task are new in WMT (Bulgarian, Basque, Dutch and Portuguese), • adaptation to the IT domain with its specifics such as frequent named entities (mostly menu items, names of prod”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “…both lead to similar results, but raters preferred continuous scales, consistent with prior findings (Svensson, 2000).14 Contrary to these findings, Bojar et al. (2016) and Novikova et al. (2018) compare direct assessments and relative rankings and find that the rankings produced were very…”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “To address the issue, the machine translation community has been organizing yearly metrics shared tasks which produce metrics that achieve a high correlation (Stanojević et al., 2015; Bojar et al., 2016, 2017; Ma et al., 2018, 2019; Mathur et al., 2020b).”
BERTScore: Evaluating Text Generation with BERT (2019, methodology): “F BERT : 481 S ENT B LEU : 256 Table 5: Examples sentences where the ranks of similarity assigned by S ENT B LEU and F BERT differ signiﬁcantly on WMT16 German-to-English reference candidate pairs. x : gold reference, ˆ x : candidate outputs of MT systems.”
BLEURT: Learning Robust Metrics for Text Generation (2020, background): “The WMT shared metrics competition (Bojar et al., 2016; Ma et al., 2018, 2019) has inspired”
How Multilingual is Multilingual BERT? (2019, methodology): “We sample 5000 pairs of sentences from WMT16 (Bojar et al., 2016) and feed each sentence (separately) to M-B ERT with no ﬁne-tuning.”
A Structured Review of the Validity of BLEU (2018, methodology): “I also automatically included all papers that presented results of the WMT conferences’ metric evaluation shared tasks, such as Bojar et al. (2016a). Colleagues have subsequently pointed out to me several relevant papers that were published after June 2017, such as Novikova et al.”
On Evaluating and Comparing Conversational Agents (2018, background): “There has also been work on learning scoring models for evaluation of MT models in WMT evaluation task (Callison-Burch et al., 2011; Bojar et al., 2016).”
Results of the WMT17 Metrics Shared Task (2017, background): “As mentioned by Bojar et al. (2016a), metrics task occasionally suffers from “loss of knowl-edge” when successful metrics participate only in one year.”
RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation (2018, methodology): “…problem of using a small dataset for training sentence embeddings. cs-en de-en ﬁ-en lv-en ro-en ru-en tr-en zh-en (Stanojevi´c et al., 2015), WMT16 (Bojar et al., 2016), and WMT17 (Bojar et al., 2017). cs-en de-en ﬁ-en lv-en ro-en ru-en tr-en zh-en WMT16 systems 6 10 9 - 7 10 8 - sentences 2,999…”
Results of the WMT20 Metrics Shared Task (2020, neutral): “To strengthen the conclusions of our evaluation, in past years (Bojar et al., 2016, 2017; Ma et al., 2018), we included signiﬁcance test results for large hybrid-super-samples of systems 10K hybrid systems were created per language pair, with corresponding DA human assessment scores by sampling…”
CharacTer: Translation Edit Rate on Character Level (2016, methodology): “Additionally we list the best performing metrics in the WMT16 metrics task (Bojar et al., 2016) in Table 3.”
chrF++: words helping character n-grams (2017, methodology): “The scores are calculated for all available translation outputs from the WMT-15 and WMT-16 shared tasks (Bojar et al., 2016) which contain two target languages, English (translated from Czech, German, Finnish, Romanian, Russian and Turkish) and Russian (translated from English), and then compared wi”
RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems (2017, background): “The Workshop on Machine Translation (WMT) organizes shared tasks for evaluation metrics (Stanojević, Kamran, and Bojar 2015; Bojar et al. 2016), attracting a large number of researchers and greatly promoting the development of translation models.”
To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation (2021, methodology): “We use the implementation of DA in the Ap-praise evaluation framework (Federmann, 2018), the same as is used in WMT since 2016 for out-of-English human evaluation (Bojar et al., 2016a).”
A Crowd-based Evaluation of Abuse Response Strategies in Conversational Agents (2019, methodology): “Following related work by (Novikova et al., 2018; Bojar et al., 2016), we use Trueskill (Herbrich et al.”
Re-evaluating Evaluation in Text Summarization (2020, methodology): “Thus, in meta-evaluation studies, calculating correlation of automatic metrics with human judgments at the system level is a commonly-used setting (Novikova et al., 2017; Bojar et al., 2016; Graham, 2015).”