Deep Exponential Families

By Rajesh Ranganath, Linpeng Tang, Laurent Charlin, David M. Blei. 2014

158 citations recorded by the source corpus.

We describe \textit{deep exponential families} (DEFs), a class of latent variable models that are inspired by the hidden structures used in deep neural networks. DEFs capture a hierarchy of dependencies between latent variables, and are easily generalized to many settings through exponential families. We perform inference using recent "black box" variational inference techniques. We then evaluate various DEFs on text and combine multiple DEFs into a model for pairwise recommendation data. In an extensive study, we show that going beyond one layer improves predictions for DEFs. We demonstrate that DEFs find interesting exploratory structure in large data sets, and give better predictive performance than state-of-the-art models.

Subfields: Computational Theory and Mathematics, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Time Series Deconfounder: Estimating Treatment Effects over Time in the Presence of Hidden Confounders (2019, methodology): “Factor Model over Time in Practice Due to the fact that we are dealing with time-varying treatment assignments, we cannot use standard factor models, such as Principal Component Analysis (Tipping & Bishop, 1999) or Deep Exponential Families (Ranganath et al., 2015), as they can only be applied in th”
Opportunities and obstacles for deep learning in biology and medicine (2017, background): “To overcome the remaining difficulties, researchers have turned to deep exponential families, a class of latent generative models that are constructed from any type of exponential family distributions [87].”
The Variational Gaussian Process (2015, methodology): “Various versions of this objective have been considered in the literature (Jaakkola & Jordan, 1998; Agakov & Barber, 2004), and it has been recently revisited by Salimans et al. (2015) and Ranganath et al. (2015b).”
Natural-Parameter Networks: A Class of Probabilistic Neural Networks (2016, background): “Thanks to the properties of the exponential family [3, 19], distributions in NPN are defined by the corresponding natural parameters which can be learned efficiently by backpropagation.”
Hierarchical Variational Models (2015, methodology): “We examine held out perplexity following the same criteria as Ranganath et al. (2015).”
The Generalized Reparameterization Gradient (2016, methodology): “We apply G - REP to perform mean-ﬁeld VI on two nonconjugate probabilistic models: the sparse gamma deep exponential family ( DEF ) and a beta-gamma matrix factorization ( MF ) model.”
Deep and Hierarchical Implicit Models (2017, background): “The former enables rich, interpretable latent structure (Neal, 1990; Ranganath et al., 2015) and applies to a wide range of tasks (Rezende et al.”
Scalable Joint Models for Reliable Uncertainty-Aware Event Prediction (2017, background): “[39] use deep exponential families [40] to develop a latent representation of diverse, multivariate data (e.”
Applications of Topic Models (2017, methodology): “Synthesis Topic modeling is also blending with more expressive latent representation models [Ranganath et al., 2015].”
The Blessings of Multiple Causes (2018, background): “A final example of a factor model is the deep exponential family (def) (Ranganath et al. 2015).”
Exponential Family Embeddings (2016, background): “…ef-emb models generalize cbow (Mikolov et al., 2013a) in the same way that exponential family pca (Collins et al., 2001) generalizes pca , glms (McCullagh and Nelder, 1989) generalize regression, and deep exponential families (Ranganath et al., 2015) generalize sigmoid belief networks (Neal, 1990).”
Factorization Meets the Item Embedding: Regularizing Matrix Factorization with Item Co-occurrence (2016, background): “[17] find that imposing a deep exponential family prior on the matrix factorization model, which implicitly conditions on consumption counts in a non-linear fashion, can be helpful in reducing the effect of extremely popular (or extremely unpopular) items on held-out recommendation tasks.”
Deep Survival Analysis (2016, methodology): “Section 3.1 reviews deep exponential families (Ranganath et al., 2015b) and Section 3.2 discusses our alignment strategy for deep survival analysis.”
A Brief Introduction to Machine Learning for Engineers (2017, methodology): “The reparametrization trick approach is for instance used in the VAE method for generative modelling [56], as well as in the black-box learning approach of [85] (see also [68]).”
Gaussian variational approximation with sparse precision matrices (2016, methodology): “They specify the prior using normalizing flows and demonstrate that the hierarchical variational model achieves better performance in terms of perplexity and held-out likelihood for deep exponential families (Ranganath et al., 2015).”
WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling (2018, background): “Deep exponential families (DEF) of Ranganath et al. (2015) construct more general probabilistic deep networks with non-binary hidden units, in which a count matrix can be factorized under the Poisson likelihood, with the gamma distributed hidden units of adjacent layers linked via the gamma scale…”
Deep Poisson Factor Modeling (2015, background): “Recently, [24] proposed a framework for generative deep models using exponential family modules.”
Auxiliary Deep Generative Models (2016, methodology): “Furthermore we have only proposed the models using a Gaussian latent distribution, but the model can easily be extended to other distributions (Ranganath et al., 2014; 2015).”
Why are deep nets reversible: A simple theory, with implications for training (2015, background): “Such models can be used to model very lifelike distributions on documents or images, and thus some of them plausibly satisfy Property (a).”