Word Translation Without Parallel Data

By Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, Herv'e J'egou. 2017

1,799 citations recorded by the source corpus.

State-of-the-art methods for learning cross-lingual word embeddings have relied on bilingual dictionaries or parallel corpora. Recent studies showed that the need for parallel data supervision can be alleviated with character-level information. While these methods showed encouraging results, they are not on par with their supervised counterparts and are limited to pairs of languages sharing a common alphabet. In this work, we show that we can build a bilingual dictionary between two languages without using any parallel corpora, by aligning monolingual word embedding spaces in an unsupervised way. Without using any character information, our model even outperforms existing supervised methods on cross-lingual tasks for some language pairs. Our experiments demonstrate that our method works very well also for distant language pairs, like English-Russian or English-Chinese. We finally describe experiments on the English-Esperanto low-resource language pair, on which there only exists a limited amount of parallel data, to show the potential impact of our method in fully unsupervised machine translation. Our code, embeddings and dictionaries are publicly available.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Cross-lingual Language Model Pretraining (2019, methodology): “Following this line of work, the need for cross-lingual supervision was further reduced (Smith et al., 2017) until it was completely removed (Conneau et al., 2018a).”
Meta-Learning for Low-Resource Neural Machine Translation (2018, methodology): “he k-th language. These embedding vectors can be obtained with small dictionaries of seed word pairs (Artetxe et al., 2017a; Smith et al., 2017) or in a fully unsupervised manner (Zhang et al., 2017; Conneau et al., 2018). We take one of these languages k0to build universal lexical representation co”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, methodology): “In this paper, we pretrain our embeddings using the common-crawl word embeddings (Grave et al., 2018) aligned with the MUSE library of Conneau et al. (2018b).”
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020, methodology): “We adopt an unsupervised word alignment method MUSE(Lample et al., 2018b), which can translate is the dictionary translating function.”
Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, background): “Cross-Lingual Representations Various previous works have explored leveraging cross-lingual word representations to build parallel dictionaries and phrase tables, then applying them to downstream tasks [1, 2, 3, 4, 31, 36].”
Unsupervised Speech Recognition (2021, methodology): “They used sequence to sequence architectures (Sutskever et al., 2014; Bahdanau et al., 2014) with shared encoder/decoders across languages, and the word embedding layer is initialized with the unsupervised bilingual word embeddings learned in Conneau et al. (2018).”
Does Object Recognition Work for Everyone? (2019, background): “Our analysis in Section 3 does suggest some approaches that may help mitigate these accuracy disparities such as geography-based resampling of image datasets and multi-lingual training of image-recognition models, for instance, via multi-lingual word embeddings [9].”
Unsupervised Machine Translation Using Monolingual Corpora Only (2017, methodology): “To keep our approach fully unsupervised, we initialize our algorithm by using a naı̈ve unsupervised translation model based on a word by word translation of sentences with a bilingual lexicon derived from the same monolingual data (Conneau et al., 2017).”
Phrase-Based & Neural Unsupervised Machine Translation (2018, methodology): “dings (Conneau et al., 2018), and e(x) is the embedding of x.”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, methodology): “…work has noted potential issues with standard datasets: MLDoc (Schwenk & Li, 2018) can be solved by matching keywords (Artetxe et al., 2019), while MultiNLI, the dataset from which XNLI (Conneau et al., 2018b) was derived, contains superﬁcial cues that can be exploited (Gururangan et al., 2018).”
Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings (2018, background): “This is proportional to the CSLS score (Conneau et al., 2018), which was originally motivated to mitigate the hub-ness problem on Bilingual Lexicon Induction (BLI) over cross-lingual word embeddings.”
When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation? (2018, methodology): “, 2017) or as a method for learning translation lexicons in an entirely unsupervised manner (Conneau et al., 2017; Gangi and Federico, 2017).”
Findings of the 2018 Conference on Machine Translation (WMT18) (2018, methodology): “Bilingual word embeddings trained in an unsupervised manner (Conneau et al., 2017) were used to translate the monolingual data by doing word-by-word translation and this synthetically created parallel data is used in the training as well.”
Unsupervised Translation of Programming Languages (2020, background): “Cross-lingual word embeddings can be obtained by training monolingual word embeddings and aligning them in an unsupervised manner [31, 6].”
On the Cross-lingual Transferability of Monolingual Representations (2019, methodology): “We describe the models that we compare ( § 3.1), the experimental setting ( § 3.2), and the results on three classiﬁcation datasets: XNLI ( § 3.3), MLDoc ( § 3.4) and PAWS-X ( § 3.5).”
FRAGE: Frequency-Agnostic Word Representation (2018, background): “[6, 22] introduce an additional discriminator to differentiate the semantics learned from different languages in non-parallel bilingual data.”
Adversarial Removal of Demographic Attributes from Text Data (2018, background): “Several works apply adversarial training to textual data, in order to learn encoders that are invariant to some properties of the text (Chen et al., 2016; Conneau et al., 2017; Zhang et al., 2017; Xie et al., 2017).”
Universal Neural Machine Translation for Extremely Low Resource Languages (2018, methodology): “However, recent efforts (Artetxe et al., 2017; Conneau et al., 2018) also showed that it is possible to learn the transformation without any seeds, which makes it feasible for our proposed method to be utilized in purely zero parallel resource cases.”