Distributed Representations of Words and Phrases and their Compositionality
By Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, Joseph Dean. Neural Information Processing Systems, 2013
35,459 citations recorded by the source corpus.
The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.
Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning
View Paper: https://arxiv.org/abs/1310.4546
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Yoshua Bengio
- Luke S. Zettlemoyer
- Quoc V. Le
- Graham Neubig
- Kyunghyun Cho
In Natural Language Processing
- Mohit Bansal
- Jianfeng Gao
- Graham Neubig
- Luke S. Zettlemoyer
- Yejin Choi
- Percy Liang
- Kyunghyun Cho
- Richard Socher
In Representation & Self-Supervised Learning
- Yoshua Bengio
- Trevor Darrell
- Dacheng Tao
- Sebastian Ruder
- Ruslan Salakhutdinov
- Andrea Vedaldi
- Quoc V. Le
- Ishan Misra
Citations in the corpus
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, methodology): “Ando and Zhang, 2005; Blitzer et al., 2006) and neural (Collobert and Weston, 2008; Mikolov et al., 2013; Pennington et al., 2014) methods.”
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014, methodology): “We have done so by applying the data selection method proposed in (Moore and Lewis, 2010), and its extension to bi-texts (Axelrod et al., 2011).”
- Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “Particularly noteworthy for NLP are algorithms for learning effective word embeddings (Collobert et al., 2011; Mikolov et al., 2013; Pennington et al., 2014) and language model pretraining (Dai and Le, 2015; Ramachandran et al.”
- Deep Contextualized Word Representations (2018, background): “Due to their ability to capture syntactic and semantic information of words from large scale unlabeled text, pretrained word vectors (Turian et al., 2010; Mikolov et al., 2013; Pennington et al., 2014) are a standard component of most state-ofthe-art NLP architectures, including for question answeri”
- An efficient framework for learning sentence representations (2018, methodology): “Alternatively, we considered an objective function similar to the negative sampling approach of Mikolov et al. (2013b).”
- Universal Language Model Fine-tuning for Text Classification (2018, methodology): “For inductive transfer, fine-tuning pretrained word embeddings (Mikolov et al., 2013), a simple transfer technique that only targets a model’s first layer, has had a large impact in practice and is used in most state-of-the-art models.”
- GloVe: Global Vectors for Word Representation (2014, methodology): “We conduct experiments on the word analogy task of Mikolov et al. (2013a), a variety of word similarity tasks, as described in (Luong et al., 2013), and on the CoNLL-2003 shared benchmark Word analogies.”
- Distributed Representations of Sentences and Documents (2014, methodology): “Our technique is inspired by the recent work in learning vector representations of words using neural networks (Bengio et al., 2006; Collobert & Weston, 2008; Mnih & Hinton, 2008; Turian et al., 2010; Mikolov et al., 2013a;c).”
- REALM: Retrieval-Augmented Language Model Pre-Training (2020, background): “Examples of this progression include models that condition on surrounding words (Mikolov et al., 2013a;b), sentences (Kiros et al., 2015; Peters et al., 2018), and paragraphs (Radford et al., 2018; Devlin et al., 2018).”
- Learning Transferable Visual Models From Natural Language Supervision (2021, background): “, 2003), word, sentence, and paragraph vectors (Mikolov et al., 2013; Kiros et al., 2015; Le & Mikolov, 2014), and language models (Bengio et al.”
- Semi-Supervised Classification with Graph Convolutional Networks (2016, background): “Recently, attention has shifted to models that learn graph embeddings with methods inspired by the skip-gram model (Mikolov et al., 2013).”