Learning Word Vectors for 157 Languages

By Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, Tomas Mikolov. 2018

1,599 citations recorded by the source corpus.

Distributed word representations, or word vectors, have recently been applied to many tasks in natural language processing, leading to state-of-the-art performance. A key ingredient to the successful application of these representations is to train them on very large corpora, and use these pre-trained models in downstream tasks. In this paper, we describe how we trained such high quality word representations for 157 languages. We used two sources of data to train these models: the free online encyclopedia Wikipedia and data from the common crawl project. We also introduce three new word analogy datasets to evaluate these word vectors, for French, Hindi and Polish. Finally, we evaluate our pre-trained word vectors on 10 languages for which evaluation datasets exists, showing very strong performance compared to previous models.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “, 2013), as a pre-training data set (Grave et al., 2018; Zellers et al., 2019; Liu et al., 2019c), and even simply as a giant text corpus for testing optimizers (Anil et al.”
Cloze-driven Pretraining of Self-attention Networks (2019, methodology): “We follow the same pre-processing as Grave et al. (2018) which is based on the May 2017 Common Crawl dump.”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, methodology): “In this paper, we pretrain our embeddings using the common-crawl word embeddings (Grave et al., 2018) aligned with the MUSE library of Conneau et al. (2018b).”
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data (2019, methodology): “More recently, Baevski et al. (2019) pre-trained a BERT-like model on Common Crawl as preprocessed in Grave et al. (2018).”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “…, such as only keeping lines with valid punctuation, discarding lines with too many symbols, or removing documents containing banned words (Grave et al., 2018; Raffel et al., 2020); (3) ML-based quality filtering , using lightweight models trained on known gold data to identify similar…”
Unsupervised Cross-lingual Representation Learning at Scale (2019, background): “to be effective for learning high quality word embeddings in multiple languages (Grave et al., 2018).”
BERTScore: Evaluating Text Generation with BERT (2019, background): “Word embeddings (Mikolov et al., 2013; Penning-ton et al., 2014; Grave et al., 2018; Nguyen et al., 2017; Athiwaratkun et al., 2018) are dense representations of tokens that are learned by optimizing an objective that follow the distributional semantics candidate, where similar words are encouraged…”
Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset (2018, methodology): “We use 300-d word embeddings pretrained on common-crawl data using fastText (Grave et al., 2018).”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, methodology): “It is deduplicated and follows the approach by (Grave et al., 2018) of using FastText LangID (Joulin et al.”
EDUCE: Explaining model Decisions through Unsupervised Concepts Extraction (2019, methodology): “We use pre-trained word vectors trained on Common Crawl [8] and keep them ﬁxed.”
Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning (2021, methodology): “; Adapter-BERT adapts the BERT as in [16], where only the adapter blocks are trainable; W2V uses embeddings trained on the Amazon review data in [66] using FastText [11].”
Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks (2021, methodology): “3); Adapter-BERT adapts the BERT as in (Houlsby et al., 2019), where only the adapter blocks are trainable; W2V uses embeddings trained on the Amazon review data in (Xu et al., 2018) using Fast-Text (Grave et al., 2018).”
What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus (2021, methodology): “The Common Crawl has been used to train many of the recent neural language models in recent years, including the GPT model series (Radford et al., 2018; Brown et al., 2020), BERT (Devlin et al., 2018) and Fast-Text (Grave et al., 2018) and, given its size, often represents the majority of data used ”
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web (2019, neutral): “5 (Grave et al., 2018).”
Mining Discourse Markers for Unsupervised Sentence Representation Learning (2019, methodology): “We keep 53% of sentence pairs that contain between 3 and 32 words, have a high probability of being English (> 75%) using FastText langid from Grave et al. (2018), have balanced parentheses and quotes, and are mostly lowercase.”
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages (2020, methodology): “Finally, as in (Grave et al., 2018), the OSCAR corpus is deduplicated, i.e. for each language, only one occurrence of a given line is included.”
CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks (2021, methodology): “Four are created using (1) BERT with fine-tuning, (2) BERT (Frozen) without finetuning (3) Adapter-BERT (Houlsby et al., 2019) and (4) W2V (word2vec embeddings trained with the Amazon review data in (Xu et al., 2018) using FastText (Grave et al., 2018)).”
AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages (2020, methodology): “We chose these hyper-parameters based on suggestions by Grave et al. (2018).”
Distilling Neural Networks for Greener and Faster Dependency Parsing (2020, methodology): “We use pretrained word embeddings from FastText (Grave et al., 2018) for all but Ancient Greek, for which we used embeddings from Ginter et al. (2017), and Wolof, for which we used embeddings from Heinzerling and Strube (2018).”