Microsoft’s Submission to the WMT2018 News Translation Task: How I Learned to Stop Worrying and Love the Data

By Marcin Junczys-Dowmunt. 2018

39 citations recorded by the source corpus.

This paper describes the Microsoft submission to the WMT2018 news translation shared task. We participated in one language direction – English-German. Our system follows current best-practice and combines state-of-the-art models with new data filtering (dual conditional cross-entropy filtering) and sentence weighting methods. We trained fairly standard Transformer-big models with an updated version of Edinburgh’s training scheme for WMT2017 and experimented with different filtering schemes for Paracrawl. According to automatic metrics (BLEU) we reached the highest score for this subtask with a nearly 2 BLEU point margin over the next strongest system. Based on human evaluation we ranked first among constrained systems. We believe this is mostly caused by our data filtering/weighting regime.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Tie-Yan Liu
3. Song Han
4. Felix Stahlberg
5. Chuang Gan
6. Yingce Xia
7. Han Cai
8. Lijun Wu

In Computer Vision and Pattern Recognition

1. Song Han
2. Tao Qin
3. Tie-Yan Liu
4. Chuang Gan
5. Felix Stahlberg
6. Quoc V. Le
7. Han Cai
8. Luc Van Gool

In Natural Language Processing

1. Tao Qin
2. Felix Stahlberg
3. Tie-Yan Liu
4. Lijun Wu
5. Yingce Xia
6. Fei Tian
7. J. Lai
8. Markus Freitag

Citations in the corpus

• The NiuTrans Machine Translation Systems for WMT19 (2019, methodology): “All parallel training data released were used and we adopted the dual conditional cross-entropy method (Junczys-Dowmunt, 2018) to filter out the noise data in ParaCrawl corpus, resulting in 10M bilingual sentences pairs.”
• Findings of the 2018 Conference on Machine Translation (WMT18) (2018, neutral): “15 MICROSOFT-MARIAN (Junczys-Dowmunt, 2018)”
• HAT: Hardware-Aware Transformers for Efficient Natural Language Processing (2020, background): “, 2017) has prevailed in sequence modeling (Ng et al., 2019; Junczys-Dowmunt, 2018).”
• Text Repair Model for Neural Machine Translation (2019, background): “Table 4 shows the results for the 2 top submissions of Microsoft (Junczys-Dowmunt, 2018) and Cambridge (Stahlberg et al., 2018).”
• Multi-Agent Dual Learning (2019, methodology): “We adopt the commercial system Google Translator (crawled on Oct 16, 2018), the champion system of WMT18 En→De: MS-Marian (Junczys-Dowmunt, 2018), and FAIR’s single and ensemble models (Edunov et al.”
• Simple and Effective Noisy Channel Modeling for Neural Machine Translation (2019, unknown): “missions (Deng et al., 2018; Koehn et al., 2018; Junczys-Dowmunt, 2018).”
• APE at Scale and Its Implications on MT Evaluation Biases (2019, methodology): “Table 4 shows the results for the 2 top submissions of Microsoft (Junczys-Dowmunt, 2018) and Cambridge (Stahlberg et al., 2018 Finally, we train our APE model on different random subsets of the available 216.5M monolingual data (see Figure 3).”
• Neural Machine Translation: A Review (2019, background): “All top systems in recent machine translation evaluation campaigns ensemble a number of NMT systems (Bojar et al., 2016, 2017, 2018, 2019; Sennrich et al., 2016a, 2017a; Neubig, 2016; Cromieres et al., 2016; Durrani et al., 2016; Stahlberg et al., 2018b; Wang et al., 2017; Junczys-Dowmunt, 2018b; Wa”
• Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation (2019, methodology): “We try to combine the strengths of last year’s two dominating systems for the English-German news translation task – FAIR’s submission with large-scale noisy back-translation (Edunov et al., 2018) and our own, based on dual cross-entropy data-filtering (Junczys-Dowmunt, 2018b,a).”
• Depth Growing for Neural Machine Translation (2019, methodology): “For example, the LSTM-based models are usually stacked for 4 (Stahlberg et al., 2018) or 6 (Chen et al., 2018) blocks, and the state-of-the-art Trans-former models are equipped with a 6 -block encoder and decoder (Vaswani et al., 2017; Junczys-Dowmunt, 2018; Edunov et al., 2018).”