Cross-lingual Models of Word Embeddings: An Empirical Comparison

By Shyam Upadhyay, Manaal Faruqui, Chris Dyer, Dan Roth. 2016

190 citations recorded by the source corpus.

Despite interest in using cross-lingual knowledge to learn word embeddings for various tasks, a systematic comparison of the possible approaches is lacking in the literature. We perform an extensive evaluation of four popular approaches of inducing cross-lingual embeddings, each requiring a different form of supervision, on four typographically different language pairs. Our evaluation setup spans four different tasks, including intrinsic evaluation on mono-lingual and cross-lingual similarity, and extrinsic evaluation on downstream semantic and syntactic applications. We show that models which require expensive cross-lingual knowledge almost always perform better, but cheaply supervised models often prove competitive on certain tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation (2019, background): “Several recent efforts have explored methods that align vector spaces for words in different languages (Faruqui and Dyer, 2014; Upadhyay et al., 2016; Ruder et al., 2017).”
Learning bilingual word embeddings with (almost) no bilingual data (2017, methodology): “In addition to the baseline systems in Section 4.2, in the crosslingual similarity experiments we also tested the method by Luong et al. (2015), which is the state-of-the-art for bilingual word embeddings based on parallel corpora (Upadhyay et al., 2016) 6 .”
Cross-Lingual Ability of Multilingual BERT: An Empirical Study (2019, background): “…shown more difﬁcult, and recent approaches relied on bilingual embeddings that allowed the transfer of supervision in high resource languages like English to models in lower resource languages; however, inducing these bilingual embeddings required some level of supervision (Upadhyay et al., 2016).”
A Survey of Cross-lingual Word Embedding Models (2017, background): “Recent work on cross-lingual embedding models suggests that the actual choice of bilingual supervision signal—that is, the data a method requires to learn to align a cross-lingual representation space—is more important for the final model performance than the actual underlying architecture (Upadhyay”
Unsupervised Cross-Lingual Representation Learning (2019, background): “Recent work has already verified the usefulness of cross-lingual word representations in a wide variety of downstream tasks, and has provided extensive model classifications in several survey papers (Upadhyay et al., 2016; Ruder et al., 2018b).”
Gromov-Wasserstein Alignment of Word Embedding Spaces (2018, background): “While the literature on this topic is extensive, we focus here on recent fully-unsupervised and minimallysupervised approaches, and refer the reader to one of various existing surveys for a broader panorama (Upadhyay et al., 2016; Ruder et al., 2017).”
Multilingual Neural Machine Translation with Task-Specific Attention (2018, background): “Recent work on learning and unsupervised induction of multilingual word representations (Upadhyay et al., 2016; Zhang et al., 2017) could also be used to improve multilingual translation since the models share a single vocabulary for all source and target languages of interest.”
Polyglot Contextual Representations Improve Crosslingual Transfer (2019, background): “State-of-the-art methods for crosslingual transfer make use of multilingual word embeddings, and much research has explored methods that align vector spaces for words in different languages (Faruqui and Dyer, 2014; Upadhyay et al., 2016; Ruder et al., 2017).”
Context-Aware Cross-Lingual Mapping (2019, background): “Cross-lingual word vector models aim to embed words from multiple languages into a shared vector space to enable cross-lingual transfer and dictionary expansion (Upadhyay et al., 2016).”
Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context (2017, background): “Incorporating crosslingual distributional information is a popular technique for learning word embeddings, and improves performance on several downstream tasks (Faruqui and Dyer, 2014; Guo et al., 2016; Upadhyay et al., 2016).”
Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations (2018, background): “As with monolingual embeddings, there exists a veritable zoo of different approaches, but they have been reported to nonetheless often perform similarly in applications (Upadhyay et al., 2016).”
Incrementally Learning the Hierarchical Softmax Function for Neural Language Models (2017, methodology): “We follow (Guo et al. 2015; Upadhyay et al. 2016) to setup the training and testing processes, using the tools available online.”
Neural Machine Translation: A Review (2019, background): “Methods for training word embeddings on raw text often take the context into account in which the word occurs frequently [41, 42], or use cross-lingual information to improve embeddings [43, 44].”
Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints (2017, background): “…vector spaces which can: a) boost performance on monolingual tasks such as word similarity (Faruqui and Dyer, 2014; Rastogi et al., 2015; Upadhyay et al., 2016); and b) support cross-lingual tasks such as bilingual lexicon induction (Mikolov et al., 2013a; Gouws et al., 2015; Duong et…”
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, background): “, 2019; Ni and Florian, 2019) with cross-lingual word embeddings (Mikolov et al., 2013a; Upadhyay et al., 2016; Ruder et al., 2019), multilingual language models have enabled a new generation”
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning (2018, background): “The use of external resources avoids the need of compiling a large parallel corpora, which has been traditionally been the main source for learning cross-lingual word embeddings in the literature (Upadhyay, Faruqui, Dyer, & Roth, 2016; Ruder, Vuli´c, & Søgaard, 2017).”
SemEval-2017 Task 2: Multilingual and Cross-lingual Semantic Word Similarity (2017, background): “Over the past few years multilingual embeddings that represent lexical items from multiple languages in a unified semantic space have garnered considerable research attention (Zou et al., 2013; de Melo, 2015; Vulić and Moens, 2016; Ammar et al., 2016; Upadhyay et al., 2016), while at the same time c”
Density Matching for Bilingual Word Embedding (2019, background): “Cross-lingual word embeddings represent words in different languages in a single vector space, capturing the syntactic and semantic similarity of words across languages in a way conducive to use in computational models (Upadhyay et al., 2016; Ruder et al., 2017).”