Offline bilingual word vectors, orthogonal transformations and the inverted softmax

By Samuel L. Smith, David H. P. Turban, Steven Hamblin, Nils Y. Hammerla. 2017

559 citations recorded by the source corpus.

Usually bilingual word vectors are trained "online". Mikolov et al. showed they can also be found "offline", whereby two pre-trained embeddings are aligned with a linear transformation, using dictionaries compiled from expert knowledge. In this work, we prove that the linear transformation between two spaces should be orthogonal. This transformation can be obtained using the singular value decomposition. We introduce a novel "inverted softmax" for identifying translation pairs, with which we improve the precision @1 of Mikolov's original mapping from 34% to 43%, when translating a test set composed of both common and rare English words into Italian. Orthogonal transformations are more robust to noise, enabling us to learn the transformation without expert bilingual signal by constructing a "pseudo-dictionary" from the identical character strings which appear in both languages, achieving 40% precision on the same test set. Finally, we extend our method to retrieve the true translations of English sentences from a corpus of 200k Italian sentences with a precision @1 of 68%.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Cross-lingual Language Model Pretraining (2019, background): “Following this line of work, the need for cross-lingual supervision was further reduced (Smith et al., 2017) until it was completely removed (Conneau et al., 2018a).”
Meta-Learning for Low-Resource Neural Machine Translation (2018, background): “These embedding vectors can be obtained with small dictionaries of seed word pairs (Artetxe et al., 2017a; Smith et al., 2017) or in a fully unsupervised manner (Zhang et al., 2017; Conneau et al., 2018).”
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020, background): “A series of followup studies focus on aligning the word representation across languages (Xing et al., 2015; Ammar et al., 2016; Smith et al., 2017; Lample et al., 2018b).”
Word Translation Without Parallel Data (2017, methodology): “We use the same embeddings as in Smith et al. (2017).”
When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation? (2018, methodology): “To test this hypothesis, we adopted the approach proposed by Smith et al. (2017) to learn orthogonal transformations that convert the word embeddings of multiple languages to a single space and used these aligned embeddings instead of independent ones.”
Similarity of Neural Network Representations Revisited (2019, methodology): “(41) In the context of neural networks, Smith et al. (2017) previously proposed using the solution to the orthogonal Procrustes problem to align word embeddings from different languages, and demonstrated that it outperformed CCA.”
Learning bilingual word embeddings with (almost) no bilingual data (2017, methodology): “While most of the previous methods use gradient descent, Artetxe et al. (2016) propose an efﬁcient analytical implementation for those same methods, recently extended by Smith et al. (2017) to incorporate dimensionality reduction.”
Universal Neural Machine Translation for Extremely Low Resource Languages (2018, background): “Since vectors are normalized, learning the optimal projection is equivalent to ﬁnding an orthogonal transformation O k that makes the projected word vectors as close as to its corresponded universal tokens: which can be solved by SVD decomposition based on the seeds (Smith et al., 2017).”
Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (2019, methodology): “transformer encoder with graph-based parser (Dozat and Manning, 2016), and dictionary supervised cross-lingual embeddings (Smith et al., 2017).”
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT (2019, background): “Our vector space alignment strategy is inspired by cross-lingual word vector alignment (e.g., Mikolov et al. (2013b); Smith et al. (2017)).”
A Survey of Cross-lingual Word Embedding Models (2017, methodology): “The vectors are available here: https://github.com/facebookresearch/fastText 10. https://github.com/Babylonpartners/fastText_multilingual As demonstrated by Smith et al. (2017), the multilingual space now enables reasoning for language pairs not represented in the seed lexicon data.”
JW300: A Wide-Coverage Parallel Corpus for Low-Resource Languages (2019, methodology): “0/ to a simple supervised method (Smith et al., 2017) in the bilingual lexicon induction task (BLI).”
Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces (2018, methodology): “Inspired by the notion, several studies have focused on designing algorithms that exploit this similarity to learn a cross-lingual alignment between the embedding spaces of two languages, where the two embedding spaces are trained from independent text corpora [6, 7, 8, 9, 10, 11, 12].”
Emerging Cross-lingual Structure in Pretrained Language Models (2019, methodology): “To measure similarity, we learn an orthogonal mapping using the Procrustes (Smith et al., 2017) approach:”
A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings (2018, methodology): “…in one or both languages under the constraint of the transformation being orthogonal (Xing et al., 2015; Artetxe et al., 2016; Zhang et al., 2016; Smith et al., 2017), and margin meth-ods, which map the embeddings in one language to maximize the margin between the correct translations and the…”
Earth Mover’s Distance Minimization for Unsupervised Bilingual Lexicon Induction (2017, methodology): “, 2016) and theoretically (Smith et al., 2017).”