Earth Mover’s Distance Minimization for Unsupervised Bilingual Lexicon Induction

By Meng Zhang, Yang Liu, Huanbo Luan, Maosong Sun. 2017

152 citations recorded by the source corpus.

Cross-lingual natural language processing hinges on the premise that there exists invariance across languages. At the word level, researchers have identified such invariance in the word embedding semantic spaces of different languages. However, in order to connect the separate spaces, cross-lingual supervision encoded in parallel data is typically required. In this paper, we attempt to establish the cross-lingual connection without relying on any cross-lingual supervision. By viewing word embedding spaces as distributions, we propose to minimize their earth mover’s distance, a measure of divergence between distributions. We demonstrate the success on the unsupervised bilingual lexicon induction task. In addition, we reveal an interesting finding that the earth mover’s distance shows potential as a measure of language difference.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Meta-Learning for Low-Resource Neural Machine Translation (2018, background): “These embedding vectors can be obtained with small dictionaries of seed word pairs (Artetxe et al., 2017a; Smith et al., 2017) or in a fully unsupervised manner (Zhang et al., 2017; Conneau et al., 2018).”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, background): “, 2018), and optimal transport (Zhang et al., 2017).”
Unsupervised Statistical Machine Translation (2018, neutral): “…dictionary, typically in the range of a few thousand entries, although a recent line of work has managed to achieve comparable results in a fully un-supervised manner based on either self-learning (Artetxe et al., 2017, 2018b) or adversarial training (Zhang et al., 2017a,b; Conneau et al., 2018).”
A Survey of Cross-lingual Word Embedding Models (2017, background): “…2016; Artetxe et al., 2017) to completely unsupervised approaches that seek to match source and target distributions based on adversarial learning (Zhang et al., 2017a; Zhang, Liu, Luan, & Sun, 2017b; Conneau et al., 2018a) and offer support to neural machine translation and cross-lingual…”
Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces (2018, background): “Motivated by the recent success in unsupervised cross-lingual alignment [13, 15, 14] and the assumption that the embedding spaces of the two modalities (speech and text) share similar structure, we are interested in learning an unsupervised cross-modal alignment between the two spaces.”
A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings (2018, methodology): “In addition to that, some datasets comprise unusually small embed-dings, with only 50 dimensions and around 5,000-10,000 vocabulary items (Zhang et al., 2017a,b).”
Density Matching for Bilingual Word Embedding (2019, methodology): “However previous work (Zhang et al., 2017) has noted that word frequency information is a strong signal – words with similar frequency are likely to be matched – and thus we use π ( y j | x i ) to incorporate this prior knowledge.”
Unsupervised Cross-lingual Word Embedding by Multilingual Neural Language Models (2018, methodology): “Zhang et al. (2017b) employ Wasserstein GAN and obtain cross-lingual representations by minimizing the earth-mover’s distance.”
One-Shot Unsupervised Cross Domain Translation (2018, methodology): “In the work on unsupervised word to word translation [9, 10, 11], the relations between words in each language are encoded by word vectors [17], and translation is well approximated by a linear transformation of one language’s vectors to those of the second.”
Context Mover's Distance & Barycenters: Optimal transport of contexts for building representations (2018, methodology): “Same also holds for barycenters, where we can compute 4,600 barycenters/second for sentences of length 25 and histogram size of 100.”
Bilingual Lexicon Induction through Unsupervised Machine Translation (2019, methodology): “At the same time, several recent methods have formulated embedding mappings as an optimal transport problem (Zhang et al., 2017b; Grave et al., 2018; Alvarez-Melis and Jaakkola, 2018).”
Gromov-Wasserstein Alignment of Word Embedding Spaces (2018, methodology): “One possible solution is to simultaneously learn an optimal coupling and an orthogonal transformation (Zhang et al., 2017b).”
Word Alignment by Fine-tuning Embeddings on Parallel Corpora (2021, background): “For example, Zhang et al. (2017) minimize the earth mover’s distance to match the embedding distributions from different languages.”
Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts (2019, methodology): “Word mover’s distance has recently been used for NLP tasks like learning word embeddings (Zhang et al., 2017; Wu et al., 2018), textual entailment (Sulea, 2017), document similarity and classiﬁcation (Kusner et al., 2015; Huang et al., 2016; Atasu et al., 2017), image captioning (Kil-ickaya et al.,…”
Generalized Data Augmentation for Low-Resource Translation (2019, background): “Mikolov et al. (2013) show that the word embedding spaces share similar innate structure over different languages, making it possible to induce bilingual dictionaries with a limited amount of or even without parallel data (Xing et al., 2015; Zhang et al., 2017; Lample et al., 2018b).”