Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation
By Marcin Junczys-Dowmunt. 2019
170 citations recorded by the source corpus.
This paper describes the Microsoft Translator submissions to the WMT19 news translation shared task for English-German. Our main focus is document-level neural machine translation with deep transformer models. We start with strong sentence-level baselines, trained on large-scale data created via data-filtering and noisy back-translation and find that back-translation seems to mainly help with translationese input. We explore fine-tuning techniques, deeper models and different ensembling strategies to counter these effects. Using document boundaries present in the authentic and synthetic parallel data, we create sequences of up to 1000 subword segments and train transformer translation models. We experiment with data augmentation techniques for the smaller authentic data with document-boundaries and for larger authentic data without boundaries. We further explore multi-task training for the incorporation of document-level source language monolingual data via the BERT-objective on the encoder and two-pass decoding for combinations of sentence-level and document-level systems. Based on preliminary human evaluation results, evaluators strongly prefer the document-level systems over our comparable sentence-level system. The document-level systems also seem to score higher than the human references in source-based direct assessment.
Subfields: Artificial Intelligence, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Graham Neubig
- Quoc V. Le
- Luke S. Zettlemoyer
- Mohit Bansal
- Kyunghyun Cho
- Yoshua Bengio
- Jianfeng Gao
- Percy Liang
In Machine Translation
- Ankur Bapna
- Orhan Firat
- Wang Ling
- Laurent Sartran
- Marta Ruiz Costa-jussà
- Angela Fan
- William Chan
- Phil Blunsom
Citations in the corpus
- Reducing Transformer Depth on Demand with Structured Dropout (2019, methodology): “This is a known problem, and techniques such as improved initialization could be used to maintain stability (Junczys-Dowmunt, 2019; Zhang et al., 2019), but are out of the scope of this work.”
- Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, background): “Corpora collected by web crawlers are known to be noisy (Junczys-Dowmunt, 2019; Caswell et al., 2020).”
- Findings of the 2019 Conference on Machine Translation (WMT19) (2019, neutral): “MS TRANSLATOR Microsoft Translator (Junczys-Dowmunt, 2019)”
- On Identifiability in Transformers (2019, neutral): “A few examples include: BERT for question answering (Alberti et al., 2019) and XL-Net (Yang et al., 2019a) with d s = 512 , or document translation (Junczys-Dowmunt, 2019) with d s = 1000 .”
- HAT: Hardware-Aware Transformers for Efficient Natural Language Processing (2020, background): “WMT’19 En-De adopts 49.6K source-target BPE vocabulary, validates on newstest2017, and tests on newstest2018, the same as Junczys-Dowmunt (2019).”
- Towards Mitigating Gender Bias in a decoder-based Neural Machine Translation model by Adding Contextual Information (2020, methodology): “The main contribution of this work is using existing NMT contextual methodologies, both context of the previous sentence (Junczys-Dowmunt, 2019) and speaker identification (Vanmassenhove et al., 2018), in a prominent and competitive NMT architecture (Fonollosa et al., 2019).”
- Putting Machine Translation in Context with the Noisy Channel Model (2019, methodology): “The idea of adding monolingual document-level data has been explored in two most recent works (Voita et al., 2019; Junczys-Dowmunt, 2019).”
- Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense (2023, methodology): “Similar efforts have also been undertaken in machine translation, [Popescu-Belis et al., 2019, Junczys-Dowmunt, 2019, Maruf et al., 2021], attempting to translate paragraphs/documents at once.”
- Big Bidirectional Insertion Representations for Documents (2019, background): “(Junczys-Dowmunt, 2019), which has been shown to improve translation quality.”
- Better Document-Level Machine Translation with Bayes’ Rule (2019, methodology): “Strategies for exploitingmonolingual documentlevel data have been explored in two recent studies (Voita et al., 2019a; Junczys-Dowmunt, 2019).”