Unsupervised Neural Machine Translation

By Mikel Artetxe, Gorka Labaka, Eneko Agirre, Kyunghyun Cho. 2017

815 citations recorded by the source corpus.

In spite of the recent success of neural machine translation (NMT) in standard benchmarks, the lack of large parallel corpora poses a major practical problem for many language pairs. There have been several proposals to alleviate this issue with, for instance, triangulation and semi-supervised learning techniques, but they still require a strong cross-lingual signal. In this work, we completely remove the need of parallel data and propose a novel method to train an NMT system in a completely unsupervised manner, relying on nothing but monolingual corpora. Our model builds upon the recent work on unsupervised embedding mappings, and consists of a slightly modified attentional encoder-decoder model that can be trained on monolingual corpora alone using a combination of denoising and backtranslation. Despite the simplicity of the approach, our system obtains 15.56 and 10.21 BLEU points in WMT 2014 French-to-English and German-to-English translation. The model can also profit from small parallel corpora, and attains 21.81 and 15.24 points when combined with 100,000 parallel sentences, respectively. Our implementation is released as an open source project.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “When no parallel data of any kind is available, Artetxe et al. (2017) and Lample et al. (2018a) proposed to jointly learn denoising auto-encoder and back-translation from both directions, which, however, required good initialization and only worked well on similar language pairs.”
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “We depart signiﬁcantly from recent unsupervised translation work (Lample et al., 2018a,b; Artetxe et al., 2018) where models are exposed to both source and target sequences, though unpaired.”
MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, methodology): “…applications such as NMT (Cho et al., 2014; Bahdanau et al., 2015a; Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017; Tan et al., 2019; Artetxe et al., 2017; Lample et al., 2017; 2018; He et al., 2018; Hassan et al., 2018; Song et al., 2018; Shen et al., 2018), text summarization…”
Cross-lingual Language Model Pretraining (2019, methodology): “Pretraining is a key ingredient of unsupervised neural machine translation (UNMT) [25, 5].”
Meta-Learning for Low-Resource Neural Machine Translation (2018, background): “The same trend can be observed even without any ﬁne-tuning (i.e., unsupervised translation, (Lam-ple et al., 2017; Artetxe et al., 2017b)).”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “For important skills, the inclusion of supervised data or task-speciﬁc data generated through unsupervised means (Artetxe et al., 2017; Lewis et al., 2019) can lead to obvious improvements.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…(Sennrich et al., 2015; Edunov et al., 2018), language model fusion (Gulcehre et al., 2015; Sriram et al., 2017), self-supervised pre-training (Dai and Le, 2015; Ramachandran et al., 2016; Zhang and Zong, 2016; Song et al., 2019) and unsupervised NMT (Lample et al., 2017; Artetxe et al., 2017).”
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (2018, background): “(Lample et al., 2017; Artetxe et al., 2017) independently proposed a denoising autoencoder in the context of sequence-to-sequence learning, where they randomly alter the word order of the input sentence and the model is trained to reconstruct the original sentence.”
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (2018, background): “(Lample et al., 2017; Artetxe et al., 2017) independently propose DAEs in the context of sequence-to-sequence learning, where they randomly alter the word order of the input sentence and the model is trained to reconstruct the original sentence.”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “Cross-lingual embeddings also provide an efficient mechanism to bootstrap neural machine translation (NMT) systems for low-resource language pairs, which is critical in the case of unsupervised machine translation (Lample et al., 2018; Artetxe et al., 2018).”
Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, background): “Unsupervised MT learns translators from unpaired corpora (Artetxe et al., 2018a;b; Lachaux et al., 2020).”
True Few-Shot Learning with Language Models (2021, background): “Other approaches assume unlabeled data and develop unsupervised model selection techniques [74–76].”
Unsupervised Question Decomposition for Question Answering (2020, methodology): “Pretraining Pretraining is crucial for unsupervised Seq2Seq methods (Artetxe et al., 2018; Lample et al., 2018), so we initialize all decomposition models (Seq2Seq or ONUS) with the same”
Unsupervised Speech Recognition (2021, background): “Going from unsupervised word translation to unsupervised sentence translation, Lample et al. (2018); Artetxe et al. (2018) exploited the similarity of neural representations across languages to build fully unsupervised machine translation systems.”
Achieving Human Parity on Automatic Chinese to English News Translation (2018, background): “Ever since dual learning was proposed, it has been successfully applied to various real-world problems such as question answering [35], image classification [47], image segmentation [25], image to image translation [50, 52, 24], face attribute manipulation [31], and machine translation [19, 43, 23, ”
Phrase-Based & Neural Unsupervised Machine Translation (2018, methodology): “Finally, our work can be seen as an extension of recent studies (Lample et al., 2018; Artetxe et al., 2018; Yang et al., 2018) on fully unsupervised MT with two major contributions.”
When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation? (2018, methodology): “org/wmt17/ translation systems (Neishi et al., 2017; Artetxe et al., 2017) or as a method for learning translation lexicons in an entirely unsupervised manner (Conneau et al.”
Findings of the 2018 Conference on Machine Translation (WMT18) (2018, methodology): “2.4.26 RWTH-UNSUPER (Graça et al., 2018) The RWTH-UNSUPER unsupervised NMT system is built based on recent works by Lample et al. (2018) and Artetxe et al. (2018).”
Unsupervised Translation of Programming Languages (2020, methodology): “More recently, several methods were proposed to train a machine translation system exclusively from monolingual corpora, using either neural models [30, 8] and statistical models [32, 7].”
Neural Architecture Optimization (2018, background): “Furthermore, the performance prediction loss acts as a regularizer that forces the encoder not optimized into a trivial state to simply copy tokens in the decoder side, which is typically eschewed by adding noise in encoding x by previous works [1, 24].”