Neural Word Embedding as Implicit Matrix Factorization

By Omer Levy, Yoav Goldberg. 2014

2,040 citations recorded by the source corpus.

Subfields: Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

context2vec: Learning Generic Context Embedding with Bidirectional LSTM (2016, methodology): “The analysis presented in Levy and Goldberg (2014b) is valid for every co-occurrence matrix that describes the joint distribution of two random variables.”
Learning Distributed Representations of Sentences from Unlabelled Data (2016, background): “We find that the optimal approach depends critically on the intended application.”
Inductive Representation Learning on Large Graphs (2017, positive): “By connection to word embedding approaches and the arguments of [20], these approaches can also be viewed as stochastic, implicit matrix factorizations where we are trying to learn a matrix where M is some matrix containing random walk statistics.”
LINE: Large-scale Information Network Embedding (2015, neutral): “We also compare with the state-of-the-art word embedding model SkipGram [12], which learns the word embeddings directly from the original Wikipedia pages and is also implicitly a matrix factorization approach [8].”
Recovery guarantee of weighted low-rank approximation via alternating minimization (2016, background): “There are many variants (e.g., [Levy and Goldberg, 2014, Pennington et al., 2014, Arora et al., 2016]); we consider the following simple approach.”
Word Translation Without Parallel Data (2017, background): “Levy & Goldberg (2014) show that the skip-gram with negative sampling method of Mikolov et al. (2013c) amounts to factorizing a word-context co-occurrence matrix, whose entries are the pointwise mutual information of the respective word and context pairs.”
Machine Learning Methods That Economists Should Know About (2019, background): “ethods. These attempt to capture latent semantic structure in language; see [Mnih and Hinton, 2007, Mnih and Teh, 2012, Mikolov et al., 2013a,b,c, Mnih and Kavukcuoglu, 2013, Pennington et al., 2014, Levy and Goldberg, 2014, Vilnis and McCallum, 2015, Arora et al., 2016, Barkan, 2016, Bamler and Man”
SHOPPER: A Probabilistic Model of Consumer Choice with Substitutes and Complements (2017, methodology): “We note that neural language models have spawned myriad developments in other so-called word embedding methods for capturing latent semantic structure in language (Mnih and Hinton (2007), Mnih and Teh (2012), Mikolov et al.”
Word Representations via Gaussian Embedding (2014, background): “Given the parallels between embedding models and matrix factorization (Deerwester et al., 1990; Riedel et al., 2013; Levy & Goldberg, 2014), this is relevant to our approach.”
Large-Scale Hierarchical Text Classification with Recursively Regularized Deep Graph-CNN (2018, methodology): “Although word2vec optimization has been proven to be identical to co-occurrence matrix factorization under mild conditions [26], it is still preferable to explicitly represent documents as graphs, since for upper level convolution, the longer distance co-occurrence of words (which corresponds convol”
Relevance-based Word Embedding (2017, background): “A theoretical relation has been discovered between embedding models based on neural network and matrix factorization in [21].”
Improving Distributional Similarity with Lessons Learned from Word Embeddings (2015, methodology): “The shift caused by k > 1 can be applied to distributional methods through shifted PPMI (Levy and Goldberg, 2014c):”
code2vec: learning distributed representations of code (2018, methodology): “An analysis by Levy and Goldberg [2014b] showed that word2vec’s skip-gram model with negative sampling is implicitly factorizing a word-context PMI matrix, linking the modern neural approaches with traditional statistical approaches.”
Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency (2018, methodology): “• We discuss application of our results to approaches of (Mnih and Teh, 2012; Mikolov et al., 2013; Levy and Goldberg, 2014; Jozefowicz et al., 2016) giving a unified account of these methods.”
What’s in an Embedding? Analyzing Word Embeddings through Multilingual Evaluation (2015, background): “Recently, word embedding techniques have been studied for their mathematical properties (Levy and Goldberg, 2014b; Stratos et al., 2015), yielding a better understanding of the underlying optimization criteria.”
A Primer on Neural Network Models for Natural Language Processing (2015, background): “The grouping is also syntactic, grouping together words that share an inflection (Levy & Goldberg, 2014a).”
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (2019, background): “Though they learn embeddings iteratively in practice, it has been proven that in theory, they both implicitly factorize a word-context matrix containing a co-occurrence statistic (Levy and Goldberg, 2014a,b).”