Learning principled bilingual mappings of word embeddings while preserving monolingual invariance

By Mikel Artetxe, Gorka Labaka, Eneko Agirre. 2016

406 citations recorded by the source corpus.

Mapping word embeddings of different languages into a single space has multiple applications. In order to map from a source space into a target space, a common approach is to learn a linear mapping that minimizes the distances between equivalences listed in a bilingual dictionary. In this paper, we propose a framework that generalizes previous work, provides an efﬁcient exact method to learn the optimal linear transformation and yields the best bilingual results in translation induction while preserving monolingual performance in an analogy task.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, background): “Cross-Lingual Representations Various previous works have explored leveraging cross-lingual word representations to build parallel dictionaries and phrase tables, then applying them to downstream tasks [1, 2, 3, 4, 31, 36].”
Unsupervised Neural Machine Translation (2017, positive): “…mapping methods independently train the embeddings in different languages using monolingual corpora, and then learn a linear transformation that maps them to a shared space based on a bilingual dictionary (Mikolov et al., 2013a; Lazaridou et al., 2015; Artetxe et al., 2016; Smith et al., 2017).”
Word Translation Without Parallel Data (2017, background): “Since then, several works aimed at improving these cross-lingual word embeddings (Faruqui & Dyer (2014); Xing et al. (2015); Baroni et al. (2015); Ammar et al. (2016); Artetxe et al. (2016); Smith et al. (2017)), but they all rely on bilingual word lexicons.”
Learning bilingual word embeddings with (almost) no bilingual data (2017, methodology): “All in all, our error analysis reveals that the baseline method of Artetxe et al. (2016) and the proposed algorithm tend to make the same kind of errors regardless of the seed dictionary used by the latter, which reinforces our interpretation in the previous section regarding an underlying…”
Offline bilingual word vectors, orthogonal transformations and the inverted softmax (2017, methodology): “A theoretical analysis of bilingual word vectors similar to this paper was recently published by Artetxe et al. (2016).”
A Survey of Cross-lingual Word Embedding Models (2017, methodology): “At this point, the most robust unsupervised method is the heuristics-based initialisation method by Artetxe et al. (2018a), while the most robust supervised method is the extension of the Procrustes method with mutual nearest neighbours by Glavas et al. (2019). We discuss challenges in Section 12.”
Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces (2018, methodology): “Inspired by the notion, several studies have focused on designing algorithms that exploit this similarity to learn a cross-lingual alignment between the embedding spaces of two languages, where the two embedding spaces are trained from independent text corpora [6, 7, 8, 9, 10, 11, 12].”
A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings (2018, methodology): “The ﬁrst two steps have been shown to be beneﬁcial in previous work (Artetxe et al., 2016), while the second length normalization guarantees the ﬁnal embeddings to have a unit length.”
Zero-shot Cross-lingual Dialogue Systems with Transferable Latent Variables (2019, background): “The goal is to ﬁnd the optimal mapping matrix W ∗ by minimizing: Following Artetxe et al. (2016), with orthogonal constraints, mean centering, and length normaliza-1 tion, we can maximize the following instead: We iteratively optimize Equation 2 until distances between domain-related seed words are…”
Unsupervised Multilingual Word Embeddings (2018, background): “It has then been observed that enforcing the linear mapping to be orthogonal could signiﬁcantly improve performance (Xing et al., 2015; Artetxe et al., 2016; Smith et al., 2017).”
Gromov-Wasserstein Alignment of Word Embedding Spaces (2018, background): “…the other, e.g., as a least-squares objective (e.g., (Mikolov et al., 2013)) or via orthogonal transformations Zhang et al. (2016); Smith et al. (2017); Artetxe et al. (2016), and methods that ﬁnd a common space on which to project both sets of embed-dings (Faruqui and Dyer, 2014; Lu et al., 2015).”
Unsupervised Cross-Lingual Representation Learning (2019, methodology): “Stochastic dictionary induction was introduced in Artetxe et al. (2018b), and we show that this bootstrapping technique improves performance and robustness, and is the main reason Artetxe et al.”
Density Matching for Bilingual Word Embedding (2019, unknown): “th the unsupervised and supervised baselines on even these difﬁcult morphologically rich languages. Supervised de-en es-en fa-en it-en Xing et al. (2015) 72 71 69 72 Shigeto et al. (2015) 72 72 69 72 Artetxe et al. (2016) 73 72 70 73 Artetxe et al. (2017) 70 70 67 71 Unsupervised Conneau et al. (201”
Unsupervised Neural Machine Translation with Weight Sharing (2018, background): “Motivated by recent success in unsupervised cross-lingual embeddings (Artetxe et al., 2016; Zhang et al., 2017b; Conneau et al., 2017), the models proposed for unsupervised NMT often assume that a pair of sentences from two different languages can be mapped to a same latent representation in a…”
Projecting Embeddings for Domain Adaption: Joint Modeling of Sentiment Analysis in Diverse Domains (2018, background): “Following state-of-the-art approaches to create bilingual word embeddings (Mikolov et al., 2013; Artetxe et al., 2016; Artetxe et al., 2017), we learn to project a mapping from a source domain vector space to the target domain space, while jointly training a sentiment classiﬁer for the source…”
Learning to Scale Multilingual Representations for Vision-Language Tasks (2020, methodology): “e highest number of languages used in multilingual learning for vision-language tasks to date, supporting more than double the number of visually-semantically aligned languages compared to prior work [5,11,16,23,36,41]. We list the contributions of our work below: { SMALR, a scalable multilingual mo”
Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations (2018, unknown): “…embeddings parallels that of word embeddings, but on a faster scale: early word embeddings models were complex and often took months to train (Bengio et al., 2003; Collobert and Weston, 2008; Turian et al., 2010) before Mikolov et al. (2013a) presented a much simpler method that could train…”
Context-Aware Cross-Lingual Mapping (2019, background): “In Artetxe et al. (2016) and Smith et al. (2017), it was shown that imposing an orthogonality constraint on the transformation leads to better word translation quality.”
Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion (2018, methodology): “Previous works (Xing et al., 2015; Artetxe et al., 2016; Smith et al., 2017) experimentally observed that constraining the mapping in such a way improves the quality of the inferred lexicon.”