Phrase-Based & Neural Unsupervised Machine Translation

By Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, Marc'Aurelio Ranzato. 2018

713 citations recorded by the source corpus.

Machine translation systems achieve near human-level performance on some languages, yet their effectiveness strongly relies on the availability of large amounts of parallel sentences, which hinders their applicability to the majority of language pairs. This work investigates how to learn to translate when having access to only large monolingual corpora in each language. We propose two model variants, a neural and a phrase-based model. Both versions leverage a careful initialization of the parameters, the denoising effect of language models and automatic generation of parallel data by iterative back-translation. These models are significantly better than methods from the literature, while being simpler and having fewer hyper-parameters. On the widely used WMT’14 English-French and WMT’16 German-English benchmarks, our models respectively obtain 28.1 and 25.2 BLEU points without using a single parallel sentence, outperforming the state of the art by more than 11 BLEU points. On low-resource languages like English-Urdu and English-Romanian, our methods achieve even better results than semi-supervised and supervised approaches leveraging the paucity of available bitexts. Our code for NMT and PBSMT is publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “When no parallel data of any kind is available, Artetxe et al. (2017) and Lample et al. (2018a) proposed to jointly learn denoising auto-encoder and back-translation from both directions, which, however, required good initialization and only worked well on similar language pairs.”
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “We depart signiﬁcantly from recent unsupervised translation work (Lample et al., 2018a,b; Artetxe et al., 2018) where models are exposed to both source and target sequences, though unpaired.”
MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, methodology): “…NMT (Cho et al., 2014; Bahdanau et al., 2015a; Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017; Tan et al., 2019; Artetxe et al., 2017; Lample et al., 2017; 2018; He et al., 2018; Hassan et al., 2018; Song et al., 2018; Shen et al., 2018), text summarization (Ayana et al., 2016;…”
Cross-lingual Language Model Pretraining (2019, methodology): “As shown in Lample et al. (2018a), this greatly improves the alignment of embedding spaces across languages that share either the same alphabet or anchor tokens such as digits (Smith et al., 2017) or proper nouns.”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, background): “For example, a criterion may perform on-the-ﬂy genera-a tion to support sequence-level training (Edunov et al., 2018b) or online backtranslation (Edunov et al., 2018a; Lample et al., 2018).”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “Cross-lingual embeddings also provide an efficient mechanism to bootstrap neural machine translation (NMT) systems for low-resource language pairs, which is critical in the case of unsupervised machine translation (Lample et al., 2018a; Artetxe et al., 2018; Lample et al., 2018b).”
Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, methodology): “The BIFI algorithm is related to backtranslation (Lample et al., 2018b) in unsupervised machine translation.”
Tagged Back-Translation (2019, methodology): “…and neural MT (G¨ulc¸ehre et al., 2015, 2017), back-translation (Sennrich et al., 2016; Poncelas et al., 2018), unsupervised machine translation (Lample et al., 2018a; Artetxe et al., 2018), dual learning (Cheng et al., 2016; Di He and Ma, 2016; Xia et al., 2017), and multi-task learning…”
Structure-to-Text Generation with Self-Training, Acceptability Classifiers and Context-Conditioning for the GEM Shared Task (2021, background): “Note that unlike work in MT that employs back-translation, including unsupervised MT (Lample et al., 2018), we do not assume access to large amounts of target text.”
The NiuTrans Machine Translation Systems for WMT19 (2019, methodology): “For unsupervised NMT, the techniques we used were based on the recently proposed for unsupervised machine translation (Lample and Conneau, 2019), including proper initialization, leveraging a strong language model and iterative backtranslation (Lample et al., 2018).”
Revisiting Self-Training for Neural Sequence Generation (2019, methodology): “We try two different perturbation functions: (1) Synthetic noise as used in unsupervised MT (Lample et al., 2018), where the input tokens are randomly dropped, masked, and shufﬂed.”
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data (2020, background): “An interesting recent development is the emergence of models for unsupervised machine translation trained only with a language modeling objective on monolingual corpora for the two languages (Lample et al., 2018).”
Billion-scale semi-supervised learning for image classification (2019, background): “On a related note, back-translation [37] is key to performance in translation [26], offering large gains by leveraging (unannotated) monolingual data.”
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond (2018, neutral): “We would also like to explore strategies to exploit monolingual data, such as using pre-trained word embeddings, back-translation (Sennrich et al., 2016; Edunov et al., 2018), or other ideas from unsupervised MT (Artetxe et al., 2018c; Lample et al., 2018).”
Findings of the 2018 Conference on Machine Translation (WMT18) (2018, methodology): “2.4.26 RWTH-UNSUPER (Graça et al., 2018) The RWTH-UNSUPER unsupervised NMT system is built based on recent works by Lample et al. (2018) and Artetxe et al. (2018).”
Unsupervised Translation of Programming Languages (2020, methodology): “[32], namely initialization, language modeling, and back-translation.”
Rapid Adaptation of Neural Machine Translation to New Languages (2018, methodology): “Finally, unsupervised NMT approaches (Artetxe et al., 2017; Lample et al., 2018, 2017) require no parallel data, but rest on strong assumptions about high-quality comparable monolingual data.”
Synthetic QA Corpora Generation with Roundtrip Consistency (2019, background): “Back-translation, which adds synthetically generated parallel data as training examples, was an inspiration for this work, and has led to state-of-the-art results in both the supervised (Edunov et al., 2018) and the unsupervised settings (Lample et al., 2018).”
The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English (2019, methodology): “Second, we consider a fully unsupervised setting, whereby only monolingual data on both the source and target side are used to train the model (Lample et al., 2018b).”
Iterative Back-Translation for Neural Machine Translation (2018, methodology): “Lample et al. (2018) explore the use of back-translated data generated by neural and statistical machine translation systems, aided by denoising with a language model trained on the target side.”