word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method

By Yoav Goldberg, Omer Levy. 2014

1,718 citations recorded by the source corpus.

The word2vec software of Tomas Mikolov and colleagues (this https URL ) has gained a lot of traction lately, and provides state-of-the-art word embeddings. The learning models behind the software are described in two research papers. We found the description of the models in these papers to be somewhat cryptic and hard to follow. While the motivations and presentation may be obvious to the neural-networks language-modeling crowd, we had to struggle quite a bit to figure out the rationale behind the equations. This note is an attempt to explain equation (4) (negative sampling) in "Distributed Representations of Words and Phrases and their Compositionality" by Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado and Jeffrey Dean.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Using the Output Embedding to Improve Language Models (2016, background): “Recently, (Goldberg and Levy, 2014) argued that the output embedding of the word2vec skip-gram model needs to be different than the input embedding.”
Knowledge Infused Learning (K-IL): Towards Deep Incorporation of Knowledge in Deep Learning (2019, methodology): “Typical approaches for such predictions employ word embeddings, such as Word2Vec, resulting in sub-optimal performance when they are used in domain-speciﬁc tasks.”
Improving Distributional Similarity with Lessons Learned from Word Embeddings (2015, background): “A detailed derivation of SGNS is available in (Goldberg and Levy, 2014).”
Dependency-Based Word Embeddings (2014, methodology): “In this section we summarize the model and training objective following the derivation presented by Goldberg and Levy (2014), and highlight the ease of incorporating arbitrary contexts in the model.”
Not All Contexts Are Created Equal: Better Word Representations with Variable Attention (2015, background): “, 2013) or resorting to negative sampling to approximate the normalizer (Goldberg and Levy, 2014).”
Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features (2017, neutral): “cantly improve training efficiency, see also (Goldberg and Levy, 2014).”
Word Embeddings Go to Italy: A Comparison of Models and Training Datasets (2015, background): “Word embeddings are typically obtained as the by-product of the training of a neural language model [6,10] that learns to predict the most probable word given a set of context words.”
metapath2vec: Scalable Representation Learning for Heterogeneous Networks (2017, background): “where Nt (v ) denotes v’s neighborhood with the t th type of nodes and p (ct |v ;θ ) is commonly dened as a somax function [3, 7, 18, 24], that is: p (ct |v;θ ) = e Xct ·Xv ∑ u∈V eXu ·Xv , where Xv is the vth row of X, representing the embedding vector for node v .”
HARP: Hierarchical Representation Learning for Networks (2017, methodology): “To solve this problem, two strategies for faster model optimization have been proposed: the hierarchical softmax and negative sampling [7].”
The Linear Representation Hypothesis and the Geometry of Large Language Models (2023, background): “…subspace hypotheses was originally observed empirically in the context of word embeddings (e.g., Mikolov et al., 2013b;c; Levy & Goldberg, 2014; Goldberg & Levy, 2014; Vylomova et al., 2016; Gladkova et al., 2016; Chiang 6 In Appendix A, we summarize these results in a figure. et al., 2020;…”
EKT: Exercise-Aware Knowledge Tracing for Student Performance Prediction (2019, methodology): “Since the intuitive one-hot representation is too sparse for modeling [14], we utilize an embedding matrix Wk ∈ RK×dk to transfer the initial knowledge encoding kt into a low-dimensional vector vt ∈ Rk with continuous values as: vt = Wkkt.”
Introduction to MAchine Learning & Knowledge Extraction (MAKE) (2017, background): “Methods to automatically encode these conceptual peculiarities emerged only recently [74,75] and open up a multitude of new business application scenarios, especially pertaining to the analysis of small snippets of text which contain insufficient information for purely statistical analysis (bag-of-w”
Large-Scale Embedding Learning in Heterogeneous Event Data (2016, background): “As [18], [19] shows, the hyperparameter of negative sampling value k [6] plays an important role in obtaining the optimal embeddings.”
The language of proteins: NLP, machine learning & protein sequences (2021, background): “words) with fixed-size dense vectors (‘‘embeddings”), such that similar words obtain similar vector representations [32].”
Neural Net Models for Open-Domain Discourse Coherence (2016, result): “The discriminative model is comparable to the negative sampling strategy (Goldberg and Levy, 2014) which has been widely used in training word embeddings.”