Polyglot: Distributed Word Representations for Multilingual NLP

By Rami Al-Rfou, Bryan Perozzi, Steven S. Skiena. 2013

496 citations recorded by the source corpus.

Distributed word representations (word embeddings) have recently contributed to competitive performance in language modeling and several NLP tasks. In this work, we train word embeddings for more than 100 languages using their corresponding Wikipedias. We quantitatively demonstrate the utility of our word embeddings by using them as the sole features for training a part of speech tagger for a subset of these languages. We find their performance to be competitive with near state-of-art methods in English, Danish and Swedish. Moreover, we investigate the semantic features captured by these embeddings through the proximity of word groupings. We will release these embeddings publicly to help researchers in the development and enhancement of multilingual applications.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Word Vectors for 157 Languages (2018, background): “A common source of data to learn word representations, available in many languages, is the online encyclopedia Wikipedia (Al-Rfou et al., 2013).”
DeepWalk: online learning of social representations (2014, background): “Distributed representations usually are trained through neural networks, these networks have made advancements in diverse fields such as computer vision [22], speech recognition [8], and natural language processing [1, 7].”
FRAGE: Frequency-Agnostic Word Representation (2018, methodology): “Following common practice [28, 1, 32, 29], we use cosine distance while computing the similarity between two word embeddings.”
BilBOWA: Fast Bilingual Distributed Representations without Word Alignments (2014, methodology): “For monolingual training data, we use the freely available, pre-tokenized Wikipedia datasets [1].”
Dependency-Based Word Embeddings (2014, background): “These representations, referred to as “neural embeddings” or “word embeddings”, have been shown to perform well across a variety of tasks (Turian et al., 2010; Collobert et al., 2011; Socher et al., 2011; Al-Rfou et al., 2013).”
Linguistic Regularities in Sparse and Explicit Word Representations (2014, background): “The effect is grouping of words that share semantic (“dog cat cow”, “eat devour”) or syntactic (“cars hats days”, “emptied carried danced”) properties, and are shown to be effective as features for various NLP tasks (Turian et al., 2010; Collobert et al., 2011; Socher et al., 2011; Al-Rfou et al., 2”
What’s in an Embedding? Analyzing Word Embeddings through Multilingual Evaluation (2015, methodology): “For English, automatically labeled data was obtained by tagging and parsing a subset of the English Wikipedia dump provided by Al-Rfou et al. (2013) using TurboTagger and TurboParser (Martins et al., 2013).”
Massively Multilingual Word Embeddings (2016, background): “Al-Rfou’ et al. (2013) trained word embeddings for more than 100 languages, but the embeddings of each language are trained independently (i.e., embeddings of words in different languages do not share the same vector space).”
Multilingual Models for Compositional Distributed Semantics (2014, methodology): “Additionally, we use the Polyglot embeddings of Al-Rfou’ et al. (2013), who published word embeddings across 100 languages, including all languages considered in this paper.”
Investigating Language Universal and Specific Properties in Word Embeddings (2016, methodology): “In this paper, C&W word vectors are all from the released version of the polyglot multilingual embeddings (Al-Rfou et al., 2013) trained on Wikipedia.”
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages (2020, methodology): “…2013), many attempts have been made to create multilingual language representations; for ﬁxed word embeddings the most remarkable works are those of (Al-Rfou et al., 2013) and (Bojanowski et al., 2017) who created word embeddings for a large quantity of languages using Wikipedia, and later (Grave…”
All-but-the-Top: Simple and Effective Postprocessing for Word Representations (2017, methodology): “In this section we explore the role of the dimension in terms of both choice ofD and the performance of the postprocessing operation – we do this by using skip-gram model on the 2010 snapshot of Wikipedia corpus (Al-Rfou et al., 2013) to train word representations.”
CamemBERT: a Tasty French Language Model (2019, background): “For non-contextual word embeddings, the ﬁrst two attempts were by (Al-Rfou et al., 2013) and (Bojanowski et al., 2017) who created word embeddings for a large number of languages using Wikipedia.”
Word Embeddings Go to Italy: A Comparison of Models and Training Datasets (2015, methodology): “Attardi has been among the firsts to produce word representations in Italian(2) by using the Italian Wikipedia, adopting the deep learning model proposed in [9], and the code(3) used in [1].”
Tailoring Continuous Word Representations for Dependency Parsing (2014, methodology): “…semantic word similarity (Huang et al., 2012; Mikolov et al., 2013a; Mikolov et al., 2013b) and used for tasks like semantic role labeling, part-of-speech tagging, NER, chunking, and sentiment classiﬁcation (Turian et al., 2010; Collobert et al., 2011; Dhillon et al., 2012; Al-Rfou’ et al., 2013).”
Unbabel’s Participation in the WMT16 Word-Level Translation Quality Estimation Shared Task (2016, methodology): “For the neural models, we used pre-trained word embeddings from Polyglot (Al-Rfou et al., 2013) and embeddings obtained from a trained neural MT system (Bahdanau et al.”
Neural Networks and Deep Learning (2018, unknown): “Multi-lingual word embeddings are presented in [9].”
Mimicking Word Embeddings using Subword RNNs (2017, methodology): “The pretrained embeddings we use in our experiments are obtained from Polyglot (Al-Rfou et al., 2013), a multilingual word embedding effort.”
Multi-Task Cross-Lingual Sequence Tagging from Scratch (2016, methodology): “For Spanish and Dutch, we use the 64-dimensional Polyglot embeddings 3 (Al-Rfou et al., 2013), which are trained on Wikipedia articles of the corresponding languages.”