Learning bilingual word embeddings with (almost) no bilingual data

By Mikel Artetxe, Gorka Labaka, Eneko Agirre. 2017

505 citations recorded by the source corpus.

Most methods to learn bilingual word embeddings rely on large parallel corpora, which is difficult to obtain for most language pairs. This has motivated an active research line to relax this requirement, with methods that use document-aligned corpora or bilingual dictionaries of a few thousand words instead. In this work, we further reduce the need of bilingual resources using a very simple self-learning approach that can be combined with any dictionary-based mapping technique. Our method exploits the structural similarity of embedding spaces, and works with as little bilingual evidence as a 25 word dictionary or even an automatically generated list of numerals, obtaining results comparable to those of systems that use richer resources.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Meta-Learning for Low-Resource Neural Machine Translation (2018, background): “The same trend can be observed even without any ﬁne-tuning (i.e., unsupervised translation, (Lam-ple et al., 2017; Artetxe et al., 2017b)).”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “More recent studies have showed that cross-lingual word embeddings can be generated with no supervision whatsoever (Artetxe et al., 2017; Conneau et al., 2018).”
Unsupervised Speech Recognition (2021, background): “Artetxe et al. (2017) greatly reduced the number of word translation pairs needed to learn the linear mapping, by iteratively refining an initial model through semi-supervised learning.”
Unsupervised Neural Machine Translation (2017, methodology): “We then use the public implementation 5 of the method proposed by Artetxe et al. (2017) to map these embeddings to a shared space, using the recommended conﬁguration with numeral-based initialization.”
Phrase-Based & Neural Unsupervised Machine Translation (2018, methodology): “(2018) used dictionaries inferred in an unsupervised way (Conneau et al., 2018; Artetxe et al., 2017).”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, background): “Over the last few years, there has been a move towards general-purpose multilingual representations that are applicable to many tasks, both on the word level (Mikolov et al., 2013; Faruqui & Dyer, 2014; Artetxe et al., 2017) or the full-sentence level (Devlin et al.”
Word Translation Without Parallel Data (2017, methodology): “For instance, the method of Artetxe et al. (2017) on our dataset does not work on the word translation task for any of the language pairs, because the digits were ﬁltered out from the datasets used to train the fastText embeddings.”
On the Cross-lingual Transferability of Monolingual Representations (2019, methodology): “, 2018a), which is further improved through self-learning (Artetxe et al., 2017).”
Universal Neural Machine Translation for Extremely Low Resource Languages (2018, methodology): “Artetxe et al. (2018) proposed a totally unsupervised approach depending on multi-lingual embedding similar to ours and duallearning and reconstruction techniques to train the model from mono-lingual data only.”
Unsupervised Statistical Machine Translation (2018, neutral): “…dictionary, typically in the range of a few thousand entries, although a recent line of work has managed to achieve comparable results in a fully un-supervised manner based on either self-learning (Artetxe et al., 2017, 2018b) or adversarial training (Zhang et al., 2017a,b; Conneau et al., 2018).”
A Survey of Cross-lingual Word Embedding Models (2017, methodology): “…for learning with less supervision have been developed: These range from approaches using small seed lexicons (Zhang, Liu, Luan, Liu, & Sun, 2016; Artetxe et al., 2017) to completely unsupervised approaches that seek to match source and target distributions based on adversarial learning (Zhang…”
Two New Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English (2019, background): “The reason for this failure is due to poor initialization: un-supervised lexicon induction techniques (Conneau et al., 2018; Artetxe et al., 2017) parallel data from ﬁltered Paracrawl improves translation quality in some conditions.”
Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces (2018, methodology): “Inspired by the notion, several studies have focused on designing algorithms that exploit this similarity to learn a cross-lingual alignment between the embedding spaces of two languages, where the two embedding spaces are trained from independent text corpora [6, 7, 8, 9, 10, 11, 12].”
Emerging Cross-lingual Structure in Pretrained Language Models (2019, background): “Follow-up efforts only required a very small seed dictionary (e.g., only numbers (Artetxe et al., 2017)) or even no dictionary at all (Conneau et al., 2017; Zhang et al., 2017).”
A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings (2018, methodology): “…in different conditions, including more challenging settings, we carry out our experiments in the widely used dataset of Dinu et al. (2015) and the subsequent extensions of Artetxe et al. (2017, 2018a), which together comprise English-Italian, English-German, English-Finnish and English-Spanish.”
Incorporating Word and Subword Units in Unsupervised Machine Translation Using Language Model Rescoring (2019, unknown): “The source and target phrases are mapped according to inferred cross-lingual word embeddings, which are trained with monolingual corpora and aligned into a shared space without any parallel data (Artetxe et al., 2017; Conneau et al., 2018).”
Are All Languages Created Equal in Multilingual BERT? (2020, background): “Previous work has found that shared strings across languages provide sufﬁcient signal for inducing cross-lingual word representations (Lample et al., 2018; Artetxe et al., 2017).”