Unsupervised Statistical Machine Translation

By Mikel Artetxe, Gorka Labaka, Eneko Agirre. 2018

260 citations recorded by the source corpus.

While modern machine translation has relied on large parallel corpora, a recent line of work has managed to train Neural Machine Translation (NMT) systems from monolingual corpora only (Artetxe et al., 2018c; Lample et al., 2018). Despite the potential of this approach for low-resource settings, existing systems are far behind their supervised counterparts, limiting their practical interest. In this paper, we propose an alternative approach based on phrase-based Statistical Machine Translation (SMT) that significantly closes the gap with supervised systems. Our method profits from the modular architecture of SMT: we first induce a phrase table from monolingual corpora through cross-lingual embedding mappings, combine it with an n-gram language model, and fine-tune hyperparameters through an unsupervised MERT variant. In addition, iterative backtranslation improves results further, yielding, for instance, 14.08 and 26.22 BLEU points in WMT 2014 English-German and English-French, respectively, an improvement of more than 7-10 BLEU points over previous unsupervised systems, and closing the gap with supervised SMT (Moses trained on Europarl) down to 2-5 BLEU points. Our implementation is available at https://github.com/artetxem/monoses.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, background): “Unsupervised MT learns translators from unpaired corpora (Artetxe et al., 2018a;b; Lachaux et al., 2020).”
Tagged Back-Translation (2019, methodology): “Recently, iterative backtranslation in both directions has been devised has a way to address unsupervised machine translation (Lample et al., 2018b; Artetxe et al., 2018).”
BPE-Dropout: Simple and Effective Subword Regularization (2019, methodology): “…techniques in natural language processing include dropping to-kens at random positions or swapping tokens at close positions (Iyyer et al., 2015; Artetxe et al., 2018; Lample et al., 2018), replacing tokens at random positions with a placeholder token (Xie et al., 2017), replacing tokens at…”
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond (2018, neutral): “…that is becoming increasingly popular is to separately train word embeddings for each language, and map them to a shared space based on a bilingual dictionary (Mikolov et al., 2013a; Artetxe et al., 2018a) or even in a fully unsupervised manner (Conneau et al., 2018a; Artetxe et al., 2018b).”
Unsupervised Translation of Programming Languages (2020, methodology): “More recently, several methods were proposed to train a machine translation system exclusively from monolingual corpora, using either neural models [30, 8] and statistical models [32, 7].”
The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English (2019, background): “, 2018) all the way to fully unsupervised approaches (Lample et al., 2018b; Artetxe et al., 2018), which use monolingual data both for learning good language models and for fantasizing parallel data.”
Unsupervised Question Answering by Cloze Translation (2019, methodology): “Most relevant to us is unsupervised NMT (Con-neau et al., 2017; Lample et al., 2017, 2018; Artetxe et al., 2018) and style transfer (Subrama-nian et al., 2018).”
The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation (2021, background): “, 2020), and even unsupervised machine translation (Lample et al., 2018; Artetxe et al., 2018) have shown promising results.”
Two New Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English (2019, background): “…2015), integration of a language model into the decoder (Gulcehre et al., 2017; Stahlberg et al., 2018) all the way to fully unsupervised approaches (Lample et al., 2018b; Artetxe et al., 2018), which use monolingual data both for learning good language models and for fantasizing parallel data.”
Text Repair Model for Neural Machine Translation (2019, methodology): “…et al., 2007) and neural MT (Gülçehre et al., 2015, 2017), back-translation (Sennrich et al., 2016b), unsupervised NMT (Lample et al., 2018a; Artetxe et al., 2018a), dual learning (Cheng et al., 2016; Di He and Ma, 2016; Yingce Xia and Liu, 2017), and multi-task learning (Domhan and…”
Density Matching for Bilingual Word Embedding (2019, methodology): “…have been shown to be an effective tool for cross-lingual NLP, e.g. the transfer of models trained on high-resource languages to low-resource ones (Klemen-tiev et al., 2012; Guo et al., 2015; Zoph et al., 2016; Zhang et al., 2018; Gu et al., 2018) or un-supervised learning (Artetxe et al., 2018c).”
A Study of Neural Matching Models for Cross-lingual IR (2020, background): “The increasing popularity of projection-based weakly-supervised [4, 6, 16] and unsupervised [1, 2] cross-lingual word embeddings has spurred unsupervised frameworks [8] for CLIR, while in the realm of mono-lingual IR, interaction-based neural matching models [5, 9, 11, 17] that utilize semantics con”
Unsupervised Parallel Sentence Extraction with Parallel Segment Detection Helps Machine Translation (2019, methodology): “In our experiments we use the NMT system of (Artetxe et al., 2018c).”
Code Translation with Compiler Representations (2022, background): “It is a simple yet powerful objective for unsupervised machine translation (Lample et al., 2018a; Artetxe et al., 2018).”
Unsupervised Cross-Lingual Representation Learning (2019, methodology): “We will put focus on a standardized two-step mappingbased framework (Artetxe et al., 2018a) that generalizes all mapping-based approaches, and analyze the importance of each component of the framework.”
Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation (2020, background): “To overcome the need for labelled data, unsupervised neural machine translation (USNMT) (Lample et al., 2018a; Artetxe et al., 2018b; Yang et al., 2018) focuses on the exploitation of very large amounts of monolingual sentences by combining denoising autoencoders with back-translation and…”
Should All Cross-Lingual Embeddings Speak English? (2019, methodology): “…is often the ﬁrst step towards several down-stream tasks such as Part-Of-Speech (POS) tagging (Zhang et al., 2016), parsing (Ammar et al., 2016a), document classiﬁcation (Klementiev et al., 2012), and machine translation (Irvine and Callison-Burch, 2013; Artetxe et al., 2018b; Lample et al., 2018).”
APE at Scale and Its Implications on MT Evaluation Biases (2019, methodology): “…(Brants et al., 2007) and neural MT (G¨ulc¸ehre et al., 2015, 2017), back-translation (Sennrich et al., 2016b), unsupervised NMT (Lample et al., 2018a; Artetxe et al., 2018a), dual learning (Cheng et al., 2016; He et al., 2016; Xia et al., 2017), and multi-task learning (Domhan and Hieber, 2017).”