A Latent Variable Model Approach to PMI-based Word Embeddings

By Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, Andrej Risteski. 2015

400 citations recorded by the source corpus.

Semantic word embeddings represent the meaning of a word via a vector, and are created by diverse methods. Many use nonlinear operations on co-occurrence statistics, and have hand-tuned hyperparameters and reweighting methods. This paper proposes a new generative model, a dynamic version of the log-linear topic model of Mnih and Hinton (2007). The methodological novelty is to use the prior to compute closed form expressions for word statistics. This provides a theoretical justification for nonlinear models like PMI, word2vec, and GloVe, as well as some hyperparameter choices. It also helps explain why low-dimensional semantic embeddings contain linear algebraic structure that allows solution of word analogies, as shown by Mikolov et al. (2013a) and many subsequent papers. Experimental support is provided for the generative model assumptions, the most important of which is that latent word vectors are fairly uniformly dispersed in space.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Feature Purification: How Adversarial Training Performs Robust Deep Learning (2020, methodology): “We use a data model called sparse coding, which is a popular model to model image, text and speech data [11, 12, 69, 77, 78, 100, 105, 106].”
Recovery guarantee of weighted low-rank approximation via alternating minimization (2016, background): “For example, the co-occurrence matrix for words in natural language processing applications (Pennington et al., 2014; Arora et al., 2016) is such that the noise is larger when the co-occurrence of two words is rarer.”
Inducing Relational Knowledge from BERT (2019, background): “While proved successful in many cases, and even somewhat supported by theoretical insights (Arora et al. 2016; Allen and Hospedales 2019), simple linear transformations of word vectors have been found limiting in more general settings (Levy and Goldberg 2014; Linzen 2016; Rogers, Drozd, and Li 2017;”
Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features (2017, methodology): “Using the generative model of (Arora et al., 2016), words are generated conditioned on a sentence “discourse” vector cs:”
Network Embedding as Matrix Factorization: Unifying DeepWalk, LINE, PTE, and node2vec (2017, neutral): “For example, Levy and Goldberg [24] prove that SGNS is actually conducting an implicit matrix factorization, which provides us with a tool to analyze the above network embedding models; Arora et al . [1] propose a generative model RAND-WALK to explain word embedding models; and Hashimoto et al . [18”
All-but-the-Top: Simple and Effective Postprocessing for Word Representations (2017, methodology): “The revised postulate, we show that the main theorem in (Arora et al., 2016) (c.f. Theorem 2.2) still holds.”
Skip-Gram − Zipf + Uniform = Vector Additivity (2017, background): “When an analogous assumption to that of (Arora et al., 2016) is made, that the context words are uniformly distributed, we prove that the composition operator reduces to vector addition.”
Unsupervised Random Walk Sentence Embeddings: A Strong but Simple Baseline (2018, methodology): “Instead of searching for the optimal hyperparameter values over some large space, as Arora et al. (2017) did, we make some simple assumptions to directly compute them.”
Large Language Models (2023, background): “One can argue [108, 5] that it follows from relations between co-occurence statistics: 30 ∀ w, M N ( w, king) / #(king) M N ( w, queen) / #(queen) ≈ M N ( w, man) / #(man) M N ( w, woman) / #(woman) (10) Given these ideas and a map F from a list of vectors to a vector, we can now propose a very…”
Linear Algebraic Structure of Word Senses, with Applications to Polysemy (2016, positive): “language (Arora et al., 2016) — a dynamic versions by the log-linear topic model of (Mnih and Hinton, 2007)—which we now recall.”
The Linear Representation Hypothesis and the Geometry of Large Language Models (2023, background): “Geometry of representations There is a line of work that studies the geometry of word and sentence representations (e.g., Arora et al., 2016; Mimno & Thompson, 2017; Etha-yarajh, 2019; Reif et al., 2019; Li et al., 2020; Hewitt & Manning, 2019; Chen et al., 2021; Chang et al., 2022; Jiang et al.,…”
Understanding Undesirable Word Embedding Associations (2019, neutral): “(Arora et al., 2016; Ethayarajh et al., 2018).”
A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors (2018, background): “, 2017), which can compete with LSTM representations, and has also been given theoretical justification as the maximum a posteriori (MAP) context vector under a generative model related to popular embedding objectives (Arora et al., 2016).”
LESS: Selecting Influential Data for Targeted Instruction Tuning (2024, background): “One possible approach is to assume a generative model for language that assign a latent topic vector to each sequence and generate tokens accordingly, as was done in Arora et al. (2016).”
Making Method of Moments Great Again? -- How can GANs learn the target distribution (2020, methodology): “ur study to other popular target distributions such as 15 (a) Wasserstein Loss (b) Parameter Distance Figure 7: Discriminators as 2-layer ReLU Networks sparse coding/non-negative matrix factorization [18, 19, 49, 47, 7] 16 9 Omitted Proofs in Section 6 9.1 Some Simple Lemmas and Properties Before we”
Humpty Dumpty: Controlling Word Meanings via Corpus Poisoning (2020, background): “Several prior approaches [7, 8, 26] derive distributional expressions for distances between word vectors, all of the form ~eu ·~ev ≈ A · log (Cu,v)−Bu −B′ v .”
On the Equivalence between Positional Node Embeddings and Structural Graph Representations (2019, background): “n, word embeddings !structural representations, where the structural equivalence between men !king and women !queen is described as being obtained by just adding or subtracting their node embeddings (Arora et al., 2016; Mikolov et al., 2013). Hence, can all node embeddings provide structural relatio”
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks (2020, result): “While some papers [Arora et al., 2016, Mu and Viswanath, 2018] have noted that word embeddings are fairly random-like in the bulk to argue that the log partition function is constant for all (cid:107) θ (cid:107) 2 = 1 , our quadratic assumption is a bit stronger.”