Advances in Pre-Training Distributed Word Representations

By Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, Armand Joulin. 2017

1,351 citations recorded by the source corpus.

Many Natural Language Processing applications nowadays rely on pre-trained word representations estimated from large text corpora such as news collections, Wikipedia and Web Crawl. In this paper, we show how to train high-quality word vector representations by using a combination of known tricks that are however rarely used together. The main result of our work is the new set of publicly available pre-trained models that outperform the current state of the art by a large margin on a number of tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, methodology): “It embeds the premise/hypothesis text using fasttext word vectors (Mikolov et al., 2018) and learned character vectors, applies a shared bidirectional LSTM to both parts, applies co-attention (Parikh et al.”
SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, methodology): “We consider the most commonly used pretrained word vectors available, namely the fastText (Mikolov et al., 2017) and the GloVe (Pennington et al., 2014) vectors trained on CommonCrawl.”
Learning Word Vectors for 157 Languages (2018, methodology): “We refer the reader to Mikolov et al. (2017) for a study of the effect of deduplication and model variants (such as position-weighted CBOW) on the quality of the word representations.”
Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function (2019, methodology): “We learn embeddings for the ACL-IMDB, RCV1, IMDB, and Arxiv datasets using word2vec (Mikolov et al. 2013) and use fastText pre-trained embeddings 2 (Mikolov et al. 2018) for all the other datasets.”
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data (2019, neutral): “Preprocessing of massive datasets for training text representations has been developed in the context of word embeddings, such as word2vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014) or fastText (Mikolov et al., 2017).”
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (2018, methodology): “We use pre-trained fast-Text word embeddings of size 300 (Mikolov et al., 2018) without ﬁne-tuning, to isolate the impact of encoder architectures and to handle words outside the training sets.”
Universal Adversarial Triggers for Attacking and Analyzing NLP (2019, methodology): “We consider Bi-LSTM models (Graves and Schmidhuber, 2005) using word2vec (Mikolov et al., 2018) or ELMo (Peters et al.”
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting (2019, methodology): “aset’s training split represented using the BOW representation. Word embeddings. The WE representation encodes the ith biography as a vector xWE i , obtained by averaging the fastText word embeddings [9, 31] for the word types present in that biography.3 The WE representation is surprisingly effecti”
When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation? (2018, methodology): “These word embeddings (Mikolov et al., 2017) incorporate character-level, phrase-level and positional information of words and are trained using CBOW algorithm (Mikolov et al.”
Do NLP Models Know Numbers? Probing Numeracy in Embeddings (2019, background): “In concurrent work, Naik et al. (2019) also explore numeracy in word vectors.”
Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases (2019, background): “fasttext (Mikolov et al., 2018) word vectors, and (5) the max of those same distances.”
What are the Biases in My Word Embedding? (2018, methodology): “, 2013), fast, trained on 600 billion words from the Web (Mikolov et al., 2018), and glove, also trained on the Web using the GloVe algorithm (Pennington, Socher, and Manning, 2014).”
EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference (2019, methodology): “2) Hypothesis-Only (HYP): FastText classifier (Mikolov et al., 2018) trained on only hypotheses”
Multilingual Constituency Parsing with Self-Attention and Pre-Training (2018, methodology): “, 2018a) outperforms ELMo, which in turn outperforms fastText (Bojanowski et al., 2017; Mikolov et al., 2018), which performs slightly better than the non pre-trained baselines.”
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages (2020, methodology): “Finally, as in (Grave et al., 2018), the OSCAR corpus is deduplicated, i.e. for each language, only one occurrence of a given line is included.”
Semantic Projection Network for Zero- and Few-Label Semantic Segmentation (2019, methodology): “In this work, we study two different word embedding models, i.e. word2vec [29] trained on Google News [47] and fastText [19] trained on Common Crawl [28].”
DeepHate: Hate Speech Detection via Multi-Faceted Text Representations (2020, methodology): “The semantic representations of social media posts are extracted from several word embeddings pre-trained on a much larger text corpus [29, 33, 43].”