Exponential Family Embeddings

By Maja R. Rudolph, Francisco J. R. Ruiz, Stephan Mandt, David M. Blei. 2016

110 citations recorded by the source corpus.

Word embeddings are a powerful approach for capturing semantic similarity among terms in a vocabulary. In this paper, we develop exponential family embeddings, a class of methods that extends the idea of word embeddings to other types of high-dimensional data. As examples, we studied neural data with real-valued observations, count data from a market basket analysis, and ratings data from a movie recommendation system. The main idea is to model each observation conditioned on a set of other observations. This set is called the context, and the way the context is defined is a modeling choice that depends on the problem. In language the context is the surrounding words; in neuroscience the context is close-by neurons; in market basket data the context is other items in the shopping cart. Each type of embedding model defines the context, the exponential family of conditional distributions, and how the latent embedding vectors are shared across data. We infer the embeddings with a scalable algorithm based on stochastic gradient descent. On all three applications—neural activity of zebrafish, users' shopping behavior, and movie ratings—we found exponential family embedding models to be more effective than other types of dimension reduction. They better reconstruct held-out data and find interesting qualitative structure.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

SHOPPER: A Probabilistic Model of Consumer Choice with Substitutes and Complements (2017, methodology): “In particular, we apply hierarchical Poisson factorization (HPF) (Gopalan, Hofman and Blei (2015)) and exponential family embeddings (Rudolph et al. (2016)). HPF focuses on user preferences; exponential family embeddings focuses on item-to-item interactions. Table 2 summarizes the differences betwee”
Dynamic Language Models for Continuously Evolving Content (2021, background): “Other works attempt to simultaneously learn time-aware embeddings over all time periods and resolve the alignment problem [4, 35, 47].”
Contextual Explanation Networks (2017, background): “Context-specific approaches are widely used in language modeling where the context is typically represented with trainable embeddings (Rudolph et al., 2016).”
Topic Modeling in Embedding Spaces (2019, background): “They have become crucial in many applications of natural language processing (Li and Tao, 2018), and they have also been extended to datasets beyond text (Rudolph et al., 2016).”
Dynamic Word Embeddings (2017, unknown): “Word embeddings are a powerful tool for modeling semantic relations between individual words (Bengio et al., 2003; Mikolov et al., 2013a; Pennington et al., 2014; Mnih & Kavukcuoglu, 2013; Levy & Goldberg, 2014; Vilnis & McCallum, 2014; Rudolph et al., 2016).”
Dynamic Word Embeddings via Skip-Gram Filtering (2017, unknown): “Word embeddings are a powerful tool for modeling semantic relations between individual words (Bengio et al., 2003; Mikolov et al., 2013a; Pennington et al., 2014; Mnih & Kavukcuoglu, 2013; Levy & Goldberg, 2014; Vilnis & Mc-Callum, 2014; Rudolph et al., 2016).”
The Geometry of Culture: Analyzing the Meanings of Class through Word Embeddings (2018, methodology): “Scientists have attempted to perform these parametrically, as with exponential family embedding models, but their performance has not yet approached that of autoencoders (Rudolph et al. 2016).”