Efficient Estimation of Word Representations in Vector Space
By Tomas Mikolov, Kai Chen, Greg S. Corrado, Joseph Dean. 2013
34,723 citations recorded by the source corpus.
We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities.
Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Percy Liang
- Luke S. Zettlemoyer
- Quoc V. Le
- Graham Neubig
- Kyunghyun Cho
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Jianfeng Gao
- Luke S. Zettlemoyer
- Yejin Choi
- Percy Liang
- Kyunghyun Cho
- Noah A. Smith
In Representation & Self-Supervised Learning
- Yoshua Bengio
- Trevor Darrell
- Ruslan Salakhutdinov
- Sebastian Ruder
- James R. Glass
- Andrew Zisserman
- Dacheng Tao
- Quoc V. Le
Citations in the corpus
- A Structured Self-attentive Sentence Embedding (2017, background): “Much progress has been made in learning semantically meaningful distributed representations of individual words, also known as word embeddings (Bengio et al., 2001; Mikolov et al., 2013).”
- Using the Output Embedding to Improve Language Models (2016, methodology): “The skip-gram word2vec model introduced in (Mikolov et al., 2013a; Mikolov et al., 2013b) learns representations of words.”
- Layer Normalization (2016, background): “Skip-thoughts [Kiros et al., 2015] is a generalization of the skip-gram model [Mikolov et al., 2013] for learning unsupervised distributed sentence representations.”
- Grammar as a Foreign Language (2014, methodology): “We pre-trained skip-gram embeddings of size 512 using word2vec [6] on a 10B-word corpus.”
- Dissecting Contextual Word Embeddings: Architecture and Representation (2018, methodology): “The word analogy task introduced in Mikolov et al. (2013) are commonly used as intrinsic evaluations of word vectors.”
- An efficient framework for learning sentence representations (2018, methodology): “Alternatively, we considered an objective function similar to the negative sampling approach of Mikolov et al. (2013b).”
- context2vec: Learning Generic Context Embedding with Bidirectional LSTM (2016, methodology): “To do this, we propose a neural network architecture, which is based on word2vec ’s CBOW architecture (Mikolov et al., 2013a), but replaces its naive context modeling of averaged word embeddings in a fixed window, with a much more powerful neural model, using bidirectional LSTM.”
- Skip-Thought Vectors (2015, methodology): “Using pretrained word2vec representations learned with a continuous bag-of-words model [8], we learn a linear mapping from a word in word2vec space to a word in the encoder’s vocabulary space.”
- Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, methodology): “The model is loosely inspired by the skip-gram [25] architecture for learning representations of words.”
- GloVe: Global Vectors for Word Representation (2014, methodology): “We conduct experiments on the word analogy task of Mikolov et al. (2013a), a variety of word similarity tasks, as described in (Luong et al., 2013), and on the CoNLL-2003 shared benchmark Word analogies.”
- Distributed Representations of Sentences and Documents (2014, methodology): “Our technique is inspired by the recent work in learning vector representations of words using neural networks (Bengio et al., 2006; Collobert & Weston, 2008; Mnih & Hinton, 2008; Turian et al., 2010; Mikolov et al., 2013a;c).”
- Distributed Representations of Words and Phrases and their Compositionality (2013, methodology): “[8] have already evaluated these word repres entations on the word analogy task, where the Skip-gram models ac hieved the best performance with a huge margin.”
- ImageNet Large Scale Visual Recognition Challenge (2014, background): “0 - - Brno University of Technology Martin Kolář, Michal Hradǐs, Pavel Svoboda (Krizhevsky et al., 2012; Mikolov et al., 2013; Jia, 2013)”
- Intriguing properties of neural networks (2013, positive): “A similar, but even stronger conclusion was reached recently by Mikolov et al. [12] for word representations, where the various directions in the vector space representing the words are shown to give rise to a surprisingly rich semantic encoding of relations and analogies.”