Skip-Gram − Zipf + Uniform = Vector Additivity

By Alex Gittens, Dimitris Achlioptas, Michael W. Mahoney. 2017

98 citations recorded by the source corpus.

In recent years word-embedding models have gained great popularity due to their remarkable performance on several tasks, including word analogy questions and caption generation. An unexpected “side-effect” of such models is that their vectors often exhibit compositionality, i.e., adding two word-vectors results in a vector that is only a small angle away from the vector of a word representing the semantic composite of the original words, e.g., “man” + “royal” = “king”. This work provides a theoretical justification for the presence of additive compositionality in word vectors learned using the Skip-Gram model. In particular, it shows that additive compositionality holds in an even stricter sense (small distance rather than small angle) under certain assumptions on the process generating the corpus. As a corollary, it explains the success of vector calculus in solving word analogies. When these assumptions do not hold, this work describes the correct non-linear composition operator. Finally, this work establishes a connection between the Skip-Gram model and the Sufficient Dimensionality Reduction (SDR) framework of Globerson and Tishby: the parameters of SDR models can be obtained from those of Skip-Gram models simply by adding information on symbol frequencies. This shows that Skip-Gram embeddings are optimal in the sense of Globerson and Tishby and, further, implies that the heuristics commonly used to approximately fit Skip-Gram models can be used to fit SDR models.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Recent Trends in Deep Learning Based Natural Language Processing (2017, background): “[20], which stated that compositionality is seen only when certain assumptions are held, e.”
Recent Trends in Deep Learning Based Natural Language Processing [Review Article] (2018, background): “[20], which stated that compositionality is seen only when certain assumptions are held, e.”
Word Embeddings and Gender Stereotypes in Swedish and English (2019, background): “Recent work [16, 10] has started studying this in the cases of GloVe and SGNS leading to the following theorem in [10].”
Linear Algebraic Structure of Word Senses, with Applications to Polysemy (2016, unknown): “, Gittens et al. (2017) and Tian et al.”
Uncovering Divergent Linguistic Information in Word Embeddings with Lessons for Intrinsic and Extrinsic Evaluation (2018, background): “While there is still an active research line to better understand these models from a theoretical perspective (Levy and Goldberg, 2014c; Arora et al., 2016; Gittens et al., 2017), the fundamental idea behind all of them is to assign a similar vector representation to similar words.”
LASAGNE: Locality and Structure Aware Graph Node Embedding (2017, methodology): “An analysis of this word2vec method has been provided by [11], reflecting a perspective similar to ours.”