Word Representations via Gaussian Embedding

By Luke Vilnis, Andrew McCallum. 2014

420 citations recorded by the source corpus.

Current work in lexical distributed representations maps each word to a point vector in low-dimensional space. Mapping instead to a density provides many interesting advantages, including better capturing uncertainty about a representation and its relationships, expressing asymmetries more naturally than dot product or cosine similarity, and enabling more expressive parameterization of decision boundaries. This paper advocates for density-based distributed embeddings and presents a method for learning representations in the space of Gaussian distributions. We compare performance on various word embedding benchmarks, investigate the ability of these embeddings to model entailment and other asymmetric relationships, and explore novel properties of the representation.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Machine Learning Methods That Economists Should Know About (2019, background): “capture latent semantic structure in language; see [Mnih and Hinton, 2007, Mnih and Teh, 2012, Mikolov et al., 2013a,b,c, Mnih and Kavukcuoglu, 2013, Pennington et al., 2014, Levy and Goldberg, 2014, Vilnis and McCallum, 2015, Arora et al., 2016, Barkan, 2016, Bamler and Mandt, 2017]. Consider the n”
SHOPPER: A Probabilistic Model of Consumer Choice with Substitutes and Complements (2017, background): “…structure in language (Mnih and Hinton, 2007; Mnih and Teh, 2012; Mikolov et al., 2013b,a; Mikolov, Yih and Zweig, 2013; Mnih and Kavukcuoglu, 2013; Pennington, Socher and Manning, 2014; Levy and Goldberg, 2014; Vilnis and McCallum, 2015; Arora et al., 2016; Barkan, 2016; Bamler and Mandt, 2017).”
Bayesian Neural Word Embedding (2016, methodology): “Bayesian methods for words representation are proposed in [8], [9].”
Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking (2017, methodology): “Unlike other approaches using Gaussian embeddings (Vilnis & McCallum, 2014; He et al., 2015; Dos Santos et al., 2016) we do not explicitly regularize the norm of the means and we do not clip the covariance matrices.”
Query2box: Reasoning over Knowledge Graphs in Vector Space using Box Embeddings (2020, background): “…words, sentences, or knowledge base concepts with geometric objects such as regions (Erk, 2009; Vilnis et al., 2018; Li et al., 2019), densities (Vilnis & McCallum, 2014; He et al., 2015; Athiwaratkun & Wilson, 2018), and orderings (Vendrov et al., 2016; Lai & Hockenmaier, 2017; Li et al.,…”
Poincaré Embeddings for Learning Hierarchical Representations (2017, background): “In relation to hierarchical representations, Vilnis and McCallum [31] proposed to learn density-based word representations, i.e., Gaussian embeddings, to capture uncertainty and asymmetry.”
Learning to Represent Knowledge Graphs with Gaussian Embedding (2015, methodology): “Moreover, we use another similarity method based on the expected likelihood or probability product kernel [28][12] to inspect the difference in performance between asymmetric and symmetric measures.”
Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry (2018, background): “In the context of word embeddings, Vilnis & McCallum (2015) proposed Gaussian Embeddings to learn improved representations.”
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning (2018, neutral): “…units (Wieting, Bansal, Gimpel, & Livescu, 2016; Bojanowski, Grave, Joulin, & Mikolov, 2017), representing words as probability distributions (Vilnis & McCallum, 2015; Athiwaratkun & Wilson, 2017; Athiwaratkun, Wilson, & Anandkumar, 2018), learning word embeddings in multilingual vector…”
Generative Models of Visually Grounded Imagination (2017, background): “However, there are also a few that use Gaussian embeddings for words (Vilnis and McCallum, 2015; Athiwaratkun and Wilson, 2017), sometimes in conjunction with images (Mukherjee and Hospedales, 2016; Ren et al.”
SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity (2016, background): “, 2011), and have some flexibility with respect to sense-level distinctions (Reisinger and Mooney, 2010b; Vilnis and McCallum, 2015).”
Order-Embeddings of Images and Language (2015, methodology): “The word2gauss baseline evaluates the approach of Vilnis & McCallum (2015) to represent words as Gaussian densities rather than points in the embedding space.”
A Dual Embedding Space Model for Document Ranking (2016, background): “Recent papers have explored in detail the SG and CBOW training methodology [11, 37] and its connection to other approaches for learning word embeddings such as explicit vector space representations [23, 24], matrix factorization [22, 33, 42] and density-based representations [45].”
Density Matching for Bilingual Word Embedding (2019, methodology): “In the future work, we will integrate Gaussian embeddings (Vilnis and McCallum, 2015) with our approach.”
Context Mover's Distance & Barycenters: Optimal transport of contexts for building representations (2018, background): “Other approaches represent words by Gaussian distributions with KL-divergence as a measure of entailment (Vilnis & McCallum, 2014; Athiwaratkun & Wilson, 2017).”
Probabilistic Face Embeddings (2019, background): “/seasonSH/ Probabilistic-Face-Embeddings the target space is explicitly deﬁned by labels. In contrast, probabilistic embeddings aim to estimate the uncertainty of the representations in latent spaces [20,37,1,28]. Speciﬁc to face recognition, some studies [7,19,51] have leveraged the model uncertain”
Deep Gaussian Embedding of Attributed Graphs: Unsupervised Inductive Learning via Ranking (2017, background): “As noted in [27] the choice of diagonal covariance coupled with the KL divergence as a similarity measure allows us to easily obtain the gradients w.”