Distributed Representations of Words and Phrases and their Compositionality

By Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, Joseph Dean. Neural Information Processing Systems, 2013

35,459 citations recorded by the source corpus.

The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

View Paper: https://arxiv.org/abs/1310.4546

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Percy Liang
  4. Yoshua Bengio
  5. Luke S. Zettlemoyer
  6. Quoc V. Le
  7. Graham Neubig
  8. Kyunghyun Cho

In Natural Language Processing

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Graham Neubig
  4. Luke S. Zettlemoyer
  5. Yejin Choi
  6. Percy Liang
  7. Kyunghyun Cho
  8. Richard Socher

In Representation & Self-Supervised Learning

  1. Yoshua Bengio
  2. Trevor Darrell
  3. Dacheng Tao
  4. Sebastian Ruder
  5. Ruslan Salakhutdinov
  6. Andrea Vedaldi
  7. Quoc V. Le
  8. Ishan Misra

Citations in the corpus