Ten Years of WMT Evaluation Campaigns: Lessons Learnt

By Ondrej Bojar, Christian Federmann, Barry Haddow, Philipp Koehn, Matt Post, Lucia Specia. 2016

39 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “…al., 2014; Bahdanau et al., 2014) have been widely adopted as the state-of-the-art approach for machine translation, both in the research community (Bojar et al., 2016a, 2017, 2018b) and for large-scale production systems (Wu et al., 2016; Zhou et al., 2016; Crego et al., 2016; Hassan et al.,…”
Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance (2018, background): “As mentioned by Bojar et al. (2016), Metrics Task occasionally suffers from “loss of knowl-edge” when successful metrics participate only in one year.”
Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges (2019, background): “In addition to validating popular metrics, including baselines metrics serves as comparison and prevents “loss of knowledge” as mentioned by Bojar et al. (2016).”
Results of the WMT17 Metrics Shared Task (2017, background): “As mentioned by Bojar et al. (2016a), metrics task occasionally suffers from “loss of knowl-edge” when successful metrics participate only in one year.”
BLEU Might Be Guilty but References Are Not Innocent (2020, background): “Interestingly, previous work (Bojar et al., 2016a) has shown that a higher correlation can be achieved when comparing similar systems than when comparing different types of systems, e.g. phrase-based vs neural vs rulebased.”
Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation (2018, methodology): “We calculate Cohen’s kappa, and speciﬁcally P ( E ) , as in WMT (Bojar et al., 2016b, Section 3.3), on the basis of all pairwise ratings across all raters.”
Best practices for the human evaluation of automatically generated text (2019, background): “In the case of largescale studies, Brysbaert (2019) recently stated that most studies with less than 50 participants are underpowered and that for most designs and analyses 100 or more participants are needed.”
Investigating Data Variance in Evaluations of Automatic Machine Translation Metrics (2022, background): “Conference Machine Translation (WMT) (Ma et al., 2019, 2018; Macháček and Bojar, 2013a,b; Bojar et al., 2016) thereby holds the Metric Shared Task to evaluate the performance of automatic metrics.”
The statistical advantage of automatic NLG metrics at the system level (2021, methodology): “In an earlier edition of WMT, DA judgments were collected with both a grammaticality prompt and an adequacy prompt, corresponding to different system rankings by the respective attribute (Bojar et al., 2016a).”