Edinburgh Neural Machine Translation Systems for WMT 16

By Rico Sennrich, Barry Haddow, Alexandra Birch. 2016

556 citations recorded by the source corpus.

We participated in the WMT 2016 shared news translation task by building neural translation systems for four language pairs, each trained in both directions: English Czech, English German, English Romanian and English Russian. Our systems are based on an attentional encoder-decoder, using BPE subword segmentation for open-vocabulary translation with a fixed vocabulary. We experimented with using automatic back-translations of the monolingual News corpus as additional training data, pervasive dropout, and target-bidirectional models. All reported methods give substantial improvements, and we see improvements of 4.3--11.2 BLEU over our baseline systems. In the human evaluation, our systems were the (tied) best constrained system for 7 out of 8 translation directions in which we participated.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Convolutional Sequence to Sequence Learning (2017, methodology): “In machine translation, this architecture has been demonstrated to outperform traditional phrase-based models by large margins (Sennrich et al., 2016b; Zhou et al., 2016; Wu et al., 2016; § 2).”
Massive Exploration of Neural Machine Translation Architectures (2017, methodology): “While some work (Luong et al., 2015b; Zhou et al., 2016; Luong and Manning, 2016; Wu et al., 2016) has achieved state of the art results using deep networks, others (Jean et al., 2015; Chung et al., 2016; Sennrich et al., 2016b) have achieved similar results with far shallower ones.”
Using the Output Embedding to Improve Language Models (2016, methodology): “State of the art results in NMT have recently been achieved by systems that segment the source and target words into subword units (Sennrich et al., 2016a).”
Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “We will include more discussions of combining Plus Back-Translation Back-translation (BT; Sennrich et al., 2016) is a standard approach to augment bi-text with target-side monolingual data.”
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2019, methodology): “We also evaluated performance on WMT16 Romanian-English, augmented with back-translation data from Sennrich et al. (2016).”
Cross-lingual Language Model Pretraining (2019, methodology): “[33] uses both back-translation and an ensemble model.”
Meta-Learning for Low-Resource Neural Machine Translation (2018, methodology): “The concatenated corpus is ﬁrst tokenized and segmented using byte-pair encoding (BPE, Sennrich et al., 2016), resulting in 40 , 000 subwords for each language.”
Deep Learning Scaling is Predictable, Empirically (2017, methodology): “We use the byte-pair encoding (BPE) method described by Sennrich et al. (2016a) to build a shared word-piece vocabulary between English and German.”
Beyond English-Centric Multilingual Machine Translation (2020, methodology): “Backtranslation (BT) creates synthetic bitexts from unaligned monolingual data (Schwenk, 2008; Bojar and Tamchyna, 2011; Sennrich et al., 2016a; Edunov et al., 2018; Hoang et al., 2018).”
The NiuTrans Machine Translation Systems for WMT19 (2019, methodology): “Back-translation (Sennrich et al., 2016b) is an essential method to integrate the target side monolingual synthetic knowledge when building a stateof-the-art NMT system.”
A Convolutional Encoder Model for Neural Machine Translation (2016, methodology): “…recurrent neural network (RNN) into a variable length representation and then generates the translation left-to-right with another RNN where both components interface via a soft-attention mechanism (Bahdanau et al., 2015; Luong et al., 2015a; Bradbury and Socher, 2016; Sennrich et al., 2016a).”
Can Neural Machine Translation be Improved with User Feedback? (2018, methodology): “The vocabulary is generated from the out-of-domain training corpus with 40k bytepair merges (Sennrich et al., 2016c) and contains 40813 source tokens and 41050 target tokens.”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “…on part of the RoBERTa corpus including Books (Zhu et al., 2015), CC-News (Nagel, 2016), OpenWebText (Gokaslan and Cohen, 2019), and CC-Stories (Trinh and Le, 2018); and (2) neural machine translation (NMT) (Ott et al., 2018) on WMT14+WMT16 (Macháˇcek and Bojar, 2014; Sennrich et al., 2016).”
8-bit Optimizers via Block-wise Quantization (2021, methodology): “We report on benchmarks in neural machine translation (Ott et al., 2018)2 trained on WMT’16 (Sennrich et al., 2016) and evaluated on en-de WMT’14 (Macháček and Bojar, 2014), large-scale language modeling (Lewis et al., 2021; Brown et al., 2020) and RoBERTa pretraining (Liu et al., 2019) on English”
Synthetic and Natural Noise Both Break Neural Machine Translation (2017, methodology): “We experimented with both their single best and ensemble BPE models, but saw no signiﬁcant difference in their performance under noise, so we report results with their single best WMT models for German/Czech → English.”
Phrase-Based & Neural Unsupervised Machine Translation (2018, background): “PBSMT is trained with true-casing, and by removing diacritics from Romanian on the source side to deal with their inconsistent use across the monolingual dataset (Sennrich et al., 2016).”