A Survey of Cross-lingual Word Embedding Models

By Sebastian Ruder, Ivan Vulic, Anders Søgaard. 2017

605 citations recorded by the source corpus.

Cross-lingual representations of words enable us to reason about word meaning in multilingual contexts and are a key facilitator of cross-lingual transfer when developing natural language processing models for low-resource languages. In this survey, we provide a comprehensive typology of cross-lingual word embedding models. We compare their data requirements and objective functions. The recurring theme of the survey is that many of the models presented in the literature optimize for the same objectives, and that seemingly different models are often equivalent, modulo optimization strategies, hyper-parameters, and such. We also discuss the different ways cross-lingual word embeddings are evaluated, as well as future challenges and research horizons.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation (2019, background): “Several recent efforts have explored methods that align vector spaces for words in different languages (Faruqui and Dyer, 2014; Upadhyay et al., 2016; Ruder et al., 2017).”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, background): “…language processing (NLP), there is a pressing urgency to build systems that serve all of the world’s approximately 6,900 languages to overcome language barriers and enable universal information access for the world’s citizens (Ruder et al., 2019; Aharoni et al., 2019; Arivazhagan et al., 2019b).”
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond (2018, neutral): “Most of this research focuses on cross-lingual word embeddings (Ruder et al., 2017), which are commonly learned jointly from parallel corpora (Gouws et al., 2015; Luong et al., 2015).”
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020, background): “…deep learning in NLP, language-speciﬁc feature design has become rare, and cross-lingual transfer methods have come into bloom (Upad-hyay et al., 2016; Ruder et al., 2019) to transfer progress from high-resourced to low-resourced languages (Adams et al., 2017; Wang et al., 2019; Kim et al., 2019).”
Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (2019, background): “Ruder et al. (2017) surveys methods for learning cross-lingual word embeddings by either joint training or post-training mappings of monolingual embeddings.”
MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer (2020, background): “…representations that can be applied to many tasks, ﬁrst on the word level (Mikolov et al., 2013; Gouws et al., 2015; Glavaˇs et al., 2019; Ruder et al., 2019; Wang et al., 2020) and later on the full-sentence level (Chidambaram et al., 2019; Devlin et al., 2019; Conneau and Lample, 2019;…”
JW300: A Wide-Coverage Parallel Corpus for Low-Resource Languages (2019, background): “Parallel data features a prominent role in building multilingual word representations (Ruder et al., 2017), annotation projection for parts-of-speech and syntactic dependencies (Das and Petrov, 2011; Tiedemann, 2014) and naturally machine translation.”
Gromov-Wasserstein Alignment of Word Embedding Spaces (2018, background): “While the literature on this topic is extensive, we focus here on recent fully-unsupervised and minimally-supervised approaches, and refer the reader to one of various existing surveys for a broader panorama (Upadhyay et al., 2016; Ruder et al., 2017).”
Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models (2023, background): “[49, 50, 51, 52, 29] Temporal Biases The training data for these models are typically restricted to limited time periods, or have temporal cutoﬀs, which may cause the model to be biased when reporting on current events, trends, and opinions.”
Bridging the domain gap in cross-lingual document classification (2019, background): “Such representations can be in the form of cross-lingual word embeddings, contextual word embeddings, or sentence embeddings (Ruder et al. (2017); Lample & Conneau (2019); Schwenk & Douze (2017)).”
Polyglot Contextual Representations Improve Crosslingual Transfer (2019, background): “State-of-the-art methods for crosslingual transfer make use of multilingual word embeddings, and much research has explored methods that align vector spaces for words in different languages (Faruqui and Dyer, 2014; Upadhyay et al., 2016; Ruder et al., 2017).”
A General Framework for Implicit and Explicit Debiasing of Distributional Word Vector Spaces (2019, methodology): “Finally, by coupling the proposed debiasing models with the cross-lingual embedding spaces (Ruder, Vuli´c, and Søgaard 2019; Glavaˇs et al. 2019), we facilitate cross-lingual debiasing transfer: we successfully debias embedding spaces in target languages without bias speciﬁcations in those…”
MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark (2020, methodology): “They also proposed various modeling techniques such as using XLU embeddings (see Ruder et al. (2017) for a review) for cross-lingual transfer and, ELMo (Peters et al., 2018) and translate train for target language training.”
Intrinsic Bias Metrics Do Not Correlate with Application Bias (2020, background): “, 2015), to extrinsic metrics, such as accuracy and F1, across multiple languages and models (Glavas et al., 2019; Ruder et al., 2019).”
Neural Unsupervised Domain Adaptation in NLP—A Survey (2020, background): “It can be seen as extreme adaptation scenario, for which parallel data may exist and can be used to build multilingual representations (Ruder et al., 2019; Artetxe et al., 2020).”