Unsupervised Machine Translation Using Monolingual Corpora Only

By Guillaume Lample, Ludovic Denoyer, Marc'Aurelio Ranzato. 2017

1,155 citations recorded by the source corpus.

Machine translation has recently achieved impressive performance thanks to recent advances in deep learning and the availability of large-scale parallel corpora. There have been numerous attempts to extend these successes to low-resource language pairs, yet requiring tens of thousands of parallel sentences. In this work, we take this research direction to the extreme and investigate whether it is possible to learn to translate even without any parallel data. We propose a model that takes sentences from monolingual corpora in two different languages and maps them into the same latent space. By learning to reconstruct in both languages from this shared feature space, the model effectively learns to translate without using any labeled data. We demonstrate our model on two widely used datasets and two language pairs, reporting BLEU scores of 32.8 and 15.1 on the Multi30k and WMT English-French datasets, without using even a single parallel sentence at training time.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “When no parallel data of any kind is available, Artetxe et al. (2017) and Lample et al. (2018a) proposed to jointly learn denoising auto-encoder and back-translation from both directions, which, however, required good initialization and only worked well on similar language pairs.”
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “We depart signiﬁcantly from recent unsupervised translation work (Lample et al., 2018a,b; Artetxe et al., 2018) where models are exposed to both source and target sequences, though unpaired.”
MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, methodology): “For NMT, we mainly investigate the zero-resource (unsupervised) setting, as unsupervised NMT has become a challenging task in recent years (Artetxe et al., 2017; Lample et al., 2017; 2018).”
Cross-lingual Language Model Pretraining (2019, methodology): “As shown in Lample et al. (2018a), this greatly improves the alignment of embedding spaces across languages that share either the same alphabet or anchor tokens such as digits (Smith et al., 2017) or proper nouns.”
Meta-Learning for Low-Resource Neural Machine Translation (2018, background): “…such as multi-task learning (Gulcehre et al., 2015; Zhang and Zong, 2016), back-translation (Sen-nrich et al., 2015), dual learning (He et al., 2016) and unsupervised machine translation with mono-lingual corpora only for both sides (Artetxe et al., 2017b; Lample et al., 2017; Yang et al., 2018).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…(Sennrich et al., 2015; Edunov et al., 2018), language model fusion (Gulcehre et al., 2015; Sriram et al., 2017), self-supervised pre-training (Dai and Le, 2015; Ramachandran et al., 2016; Zhang and Zong, 2016; Song et al., 2019) and unsupervised NMT (Lample et al., 2017; Artetxe et al., 2017).”
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (2018, background): “(Lample et al., 2017; Artetxe et al., 2017) independently proposed a denoising autoencoder in the context of sequence-to-sequence learning, where they randomly alter the word order of the input sentence and the model is trained to reconstruct the original sentence.”
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (2018, background): “(Lample et al., 2017; Artetxe et al., 2017) independently propose DAEs in the context of sequence-to-sequence learning, where they randomly alter the word order of the input sentence and the model is trained to reconstruct the original sentence.”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “Cross-lingual embeddings also provide an efficient mechanism to bootstrap neural machine translation (NMT) systems for low-resource language pairs, which is critical in the case of unsupervised machine translation (Lample et al., 2018; Artetxe et al., 2018).”
Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, methodology): “The BIFI algorithm is related to backtranslation (Lample et al., 2018b) in unsupervised machine translation.”
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020, methodology): “We adopt an unsupervised word alignment method MUSE(Lample et al., 2018b), which can translate is the dictionary translating function.”
Tagged Back-Translation (2019, methodology): “…and neural MT (G¨ulc¸ehre et al., 2015, 2017), back-translation (Sennrich et al., 2016; Poncelas et al., 2018), unsupervised machine translation (Lample et al., 2018a; Artetxe et al., 2018), dual learning (Cheng et al., 2016; Di He and Ma, 2016; Xia et al., 2017), and multi-task learning…”
Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, background): “A major line of work that does not use any labeled parallel data typically works as follow: They ﬁrst train an initial language model using a noisy reconstruction objective, then ﬁnetune it using on-the-ﬂy backtranslation loss [16, 32, 19, 18, 8].”
Parallel Corpus Filtering via Pre-trained Language Models (2020, background): “A similar observation was made in Lample et al. (2018), where the cross-lingual word embeddings learned in an unsupervised manner are loosely aligned.”
Unsupervised Question Decomposition for Question Answering (2020, methodology): “To stop training without supervision, we use a modiﬁed version of round-trip BLEU (Lample et al., 2018) (see Appendix § B.1 for details).”
Unsupervised Speech Recognition (2021, background): “Going from unsupervised word translation to unsupervised sentence translation, Lample et al. (2018); Artetxe et al. (2018) exploited the similarity of neural representations across languages to build fully unsupervised machine translation systems.”
BPE-Dropout: Simple and Effective Subword Regularization (2019, background): “…language processing include dropping to-kens at random positions or swapping tokens at close positions (Iyyer et al., 2015; Artetxe et al., 2018; Lample et al., 2018), replacing tokens at random positions with a placeholder token (Xie et al., 2017), replacing tokens at random positions with a…”
Achieving Human Parity on Automatic Chinese to English News Translation (2018, background): “Ever since dual learning was proposed, it has been successfully applied to various real-world problems such as question answering [35], image classification [47], image segmentation [25], image to image translation [50, 52, 24], face attribute manipulation [31], and machine translation [19, 43, 23, ”