Findings of the 2018 Conference on Machine Translation (WMT18)

By Ondrej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, Christof Monz. 2018

397 citations recorded by the source corpus.

This paper presents the results of the premier shared task organized alongside the Conference on Machine Translation (WMT) 2018. Participants were asked to build machine translation systems for any of 7 language pairs in both directions, to be evaluated on a test set of news stories. The main metric for this task is human judgment of translation quality. This year, we also opened up the task to additional test sets to probe specific aspects of translation.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Massively Multilingual Neural Machine Translation (2019, methodology): “For comparison, this is signiﬁcantly “more multilingual” than the data available from all previous WMT news translation shared task evaluations throughout the years – the latest being Bojar et al. (2016, 2017, 2018), which included 14 languages so far.”
The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (2019, background): “…et al., 2017), as this architecture achieves state-of-the-art results on tasks with very different objectives such as machine translation (MT) (Bojar et al., 2018; Niehues et al., 2018), standard left-to-right language modeling (LM) (Radford et al., 2018) and masked language modeling (MLM)…”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “…al., 2014; Bahdanau et al., 2014) have been widely adopted as the state-of-the-art approach for machine translation, both in the research community (Bojar et al., 2016a, 2017, 2018b) and for large-scale production systems (Wu et al., 2016; Zhou et al., 2016; Crego et al., 2016; Hassan et al.,…”
Cloze-driven Pretraining of Self-attention Networks (2019, methodology): “We use up to 4.5B words of English news web data distributed as part of WMT 2018 (Bojar et al., 2018).”
Beyond English-Centric Multilingual Machine Translation (2020, neutral): “We consider data for 13 languages (Ondrej et al., 2017; Bojar et al., 2018; Barrault et al., 2019).”
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (2019, background): “The Transformer achieved state-of-the-art results in recent shared translation tasks (Bojar et al., 2018; Niehues et al., 2018).”
mSLAM: Massively multilingual joint pre-training for speech and text (2022, neutral): “, 2019) for de, ru, zh, WMT18 (Bojar et al., 2018) for et, tr, WMT17 (Bojar et al.”
BPE-Dropout: Simple and Effective Subword Regularization (2019, methodology): “Using subword segmentation has become de-facto standard in Neural Machine Translation (Bojar et al., 2018; Barrault et al., 2019).”
Nearest Neighbor Machine Translation (2020, methodology): “This is a highly competitive baseline – apart from the WMT’19 training data, the base model has also been trained on over 10 billion tokens of extra backtranslation data as well as ﬁne-tuned on newstest test sets from previous years.”
Findings of the 2019 Conference on Machine Translation (WMT19) (2019, background): “It is thus not clear if the super-human quality observed last year was due to lower quality of last year’s references, different set of documents or the segment-level style of evaluation as thoroughly discussed by Bojar et al. (2018).”
The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation (2021, background): “, 2017), Kazakh in 2018 (Bojar et al., 2018), Gujarati and Lithuanian in 2019 (Barrault et al.”
BLEU Might Be Guilty but References Are Not Innocent (2020, neutral): “Similarly, recent benchmark results comparing neural systems on high resource languages (Bojar et al., 2018; Barrault et al., 2019) have shown mismatches between the systems with highest BLEU score and the systems faring the best in human evaluations.”
Multilingual Translation from Denoising Pre-Training (2021, methodology): “All of the data is publicly available, such as WMT (Bojar et al., 2013, 2014, 2016, 2017, 2018; Barrault et al., 2019, 2020), IWSLT (Luong and Manning, 2015; Cettolo et al., 2017), TED58 (Qi et al., 2018), OPUS (Tiedemann, 2012), WAT (Nakazawa et al., 2019), LauraMarti-nus (Abbott and Martinus, 2019”
DeepNet: Scaling Transformers to 1,000 Layers (2022, methodology): “Following M2M-100 [24], we evaluate the models on several multilingual translation evaluation data sets, including WMT [39], [40], [41], [42], OPUS [33], TED [43], and Flo-res [44].”
Context Mover's Distance & Barycenters: Optimal transport of contexts for building representations (2018, methodology): “(iii) Overall comparisons: To facilitate an accurate comparison with baselines which typically use huge corporas, in Table 1 (c) we report our results (with ground metric GloVe) by using the News Crawl corpus (Bojar et al., 2018) combined with TBC.”
Automatic Text Evaluation through the Lens of Wasserstein Barycenters (2021, background): “, 2019) and choose to work with WMT16 and additionally report results on WMT15 that both offer over 500 sentences per language (in contrast to new versions (Bojar et al., 2018; Barrault et al., 2019, 2020) that rely on a lower number–around 50–of annotated texts).”