Massively Multilingual Word Embeddings

By Bridger Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, Noah A. Smith. 2016

276 citations recorded by the source corpus.

We introduce new methods for estimating and evaluating embeddings of words in more than fifty languages in a single shared embedding space. Our estimation methods, multiCluster and multiCCA, use dictionaries and monolingual data; they do not require parallel data. Our new evaluation method, multiQVEC-CCA, is shown to correlate better than previous ones with two downstream tasks (text categorization and parsing). We also describe a web portal for evaluation that will facilitate further research in this area, along with open-source releases of all our methods.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Cross-lingual Language Model Pretraining (2019, background): “A series of follow-up studies show that cross-lingual representations can be used to improve the quality of monolingual representations [16], that orthogonal transformations are sufficient to align these word distributions [43], and that all these techniques can be applied to an arbitrary number of ”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “Many of these methods require some form of supervision (typically in the form of a small bilingual lexicon) to align two sets of source and target embeddings to the same space (Mikolov et al., 2013a; Kociský et al., 2014; Faruqui and Dyer, 2014; Ammar et al., 2016).”
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020, background): “A series of followup studies focus on aligning the word representation across languages (Xing et al., 2015; Ammar et al., 2016; Smith et al., 2017; Lample et al., 2018b).”
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond (2018, neutral): “To the best of our knowledge, existing work on learning multilingual representations for a large number of languages is limited to word embeddings (Ammar et al., 2016; Dufter et al., 2018) specific applications like typology prediction (Malaviya et al., 2017) or machine translation (Neubig and Hu,…”
Word Translation Without Parallel Data (2017, background): “Since then, several works aimed at improving these cross-lingual word embeddings (Faruqui & Dyer (2014); Xing et al. (2015); Baroni et al. (2015); Ammar et al. (2016); Artetxe et al. (2016); Smith et al. (2017)), but they all rely on bilingual word lexicons.”
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks (2019, methodology): “Following the line of previous work, the word alignment also could be applied to multiple languages (Ammar et al., 2016).”
Offline bilingual word vectors, orthogonal transformations and the inverted softmax (2017, background): “…matrix should be orthogonal, and introduced an approximate procedure composed of gradient descent updates and repeated applications of the SVD. CCA has been extended to map 59 languages into a single shared space (Ammar et al., 2016), and non-linear “deep CCA” has been introduced (Lu et al., 2015).”
Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (2019, neutral): “Schwenk and Li (2018) use MultiCCA, multilingual word embeddings trained with a bilingual dictionary (Ammar et al., 2016), and convolution neural networks.”
A Corpus for Multilingual Document Classification in Eight Languages (2018, methodology): “To provide reproducible benchmark results, we use MultiCCA word embeddings published by (Ammar et al., 2016).”
A Survey of Cross-lingual Word Embedding Models (2017, methodology): “Despite its obvious simplicity, the only work that generalizes pseudo-bilingual approaches to the multilingual setting that we are aware of is by Ammar et al. (2016b) who replace all tokens in monolingual corpora with their corresponding translation cluster ID, thereby restricting them to have the…”
Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus (2020, background): “Recently, however, signiﬁcant progress has been made in cross-lingual learning for NLP tasks (Kle-mentiev et al., 2012; Ammar et al., 2016; Lample and Conneau, 2019; Pfeiffer et al., 2020): for example, some approaches appear capable of extending machine translation models to new languages with…”
Challenges and Strategies in Cross-Cultural NLP (2022, background): “In this context, approaches for creating crosslingual word embeddings (Klementiev et al., 2012; Ammar et al., 2016) are based on the assumption that the semantic spaces of different languages are approximately isomorphic.”
Neural Cross-Lingual Entity Linking (2017, methodology): “MultiCCA: Introduced by (Ammar et al. 2016) this technique builds upon CCA and uses a linear operator to project pre-trained monolingual embeddings in each language (except English) to the vector space of pre-trained English word embeddings.”
Learning to Actively Learn Neural Machine Translation (2018, methodology): “In our experiments, we either use the pre-trained word embeddings from (Ammar et al., 2016) or build it based on the available bitext and monotext in the source and target language (c.f. § parameter 5.2).”
Neural Sequence Learning Models for Word Sense Disambiguation (2017, background): “…with two different conﬁgurations of the embedding layer: the pre-trained bilingual embeddings by Mrkˇsi´c et al. (2017) for all the language pairs of interest (EN-IT, EN-FR, EN-DE, and EN-ES), and the pre-trained multilingual 512-dimensional embeddings for 12 languages by Ammar et al. (2016).”
Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer (2020, background): “Multilingual word embeddings align the embeddings from various languages to the same shared embedding space which enables transfer learning by training the model in one language and adopting it for another one (Ammar et al., 2016; Ahmad et al., 2019b; Meng et al., 2019; Chen et al., 2019).”
SemEval-2017 Task 2: Multilingual and Cross-lingual Semantic Word Similarity (2017, background): “…multiple languages in a uniﬁed semantic space have garnered considerable research attention (Zou et al., 2013; de Melo, 2015; Vuli´c and Moens, 2016; Ammar et al., 2016; Upadhyay et al., 2016), while at the same time cross-lingual applications have also been increasingly studied (Xiao and Guo,…”
Unsupervised Cross-Lingual Representation Learning (2019, background): “In addition, they are a key facilitator of cross-lingual transfer and joint multilingual training, offering support to NLP applications in a large spectrum of languages (Søgaard et al., 2015; Ammar et al., 2016a).”