word2vec Parameter Learning Explained

By Xin Rong. 2014

885 citations recorded by the source corpus.

The word2vec model and application by Mikolov et al. have attracted a great amount of attention in recent two years. The vector representations of words learned by word2vec models have been shown to carry semantic meanings and are useful in various NLP tasks. As an increasing number of researchers would like to experiment with word2vec or similar techniques, I notice that there lacks a material that comprehensively explains the parameter learning process of word embedding models in details, thus preventing researchers that are non-experts in neural networks from understanding the working mechanism of such models. This note provides detailed derivations and explanations of the parameter update equations of the word2vec models, including the original continuous bag-of-word (CBOW) and skip-gram (SG) models, as well as advanced optimization techniques, including hierarchical softmax and negative sampling. Intuitive interpretations of the gradient equations are also provided alongside mathematical derivations. In the appendix, a review on the basics of neuron networks and backpropagation is provided. I also created an interactive demo, wevi, to facilitate the intuitive understanding of the model.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Recent Trends in Deep Learning Based Natural Language Processing (2017, background): “4: Model for CBOW (Figure source: Rong [22])”
A Primer on Neural Network Models for Natural Language Processing (2015, background): “See work by Rong (2014) and Levy, Goldberg, and Dagan (2015) for a discussion.”
Introduction to MAchine Learning & Knowledge Extraction (MAKE) (2017, background): “Methods to automatically encode these conceptual peculiarities emerged only recently [74,75] and open up a multitude of new business application scenarios, especially pertaining to the analysis of small snippets of text which contain insufficient information for purely statistical analysis (bag-of-w”
Radical-Based Hierarchical Embeddings for Chinese Sentiment Analysis at Sentence Level (2017, background): “Speciﬁcally, it is a one hidden layer neural network (Rong 2014).”
Misinformation in Social Media: Definition, Manipulation, and Detection (2019, methodology): “Therefore, in contrast to using raw words, various methods test using latent semantic features that are learned from words, such as Language Modeling (LM) [28], word embeddings [33] and Topic Modeling [4].”
Deep Learning Based Vulnerability Detection: Are We There Yet? (2020, background): “Another option is to use external word embedding tool( e.g., Word2Vec [33]) to create vector representation of every token.”
TabTransformer: Tabular Data Modeling Using Contextual Embeddings (2020, methodology): “Beginning from Word2Vec (Rong 2014) with the context-free word embeddings to BERT (Devlin et al.”
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers (2018, positive): “Some examples compute the saliency for groups of words across the model and visualize the values as a matrix [41], while others also use matrix visualization to show the activations of word groups when represented as feature vectors in a word embedding [50], [83].”
Text Classification Algorithms: A Survey (2019, background): “The weights between the input layer and output layer represent v× N [63] as a matrix of w.”
A Dual Embedding Space Model for Document Ranking (2016, background): “Recent papers have explored in detail the SG and CBOW training methodology [11, 37] and its connection to other approaches for learning word embeddings such as explicit vector space representations [23, 24], matrix factorization [22, 33, 42] and density-based representations [45].”
Neural Networks and Deep Learning (2018, neutral): “The word2vec model is proposed in [325, 327], and a detailed exposition may be found in [404].”
metapath2vec: Scalable Representation Learning for Heterogeneous Networks (2017, background): “where Nt (v ) denotes v’s neighborhood with the t th type of nodes and p (ct |v ;θ ) is commonly dened as a somax function [3, 7, 18, 24], that is: p (ct |v;θ ) = e Xct ·Xv ∑ u∈V eXu ·Xv , where Xv is the vth row of X, representing the embedding vector for node v .”