Exponential Family Harmoniums with an Application to Information Retrieval

By Max Welling, Michal Rosen-Zvi, Geoffrey E. Hinton. 2004

522 citations recorded by the source corpus.

Subfields: Information Retrieval & Search, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Retrieval & Search

In Probabilistic & Bayesian Methods

Citations in the corpus

Pylearn2: a machine learning research library (2013, methodology): “This includes autoencoders [6], RBMs [47] including RBMs with Ga ussian visible units [54], DBMs [45], MLPs [44], convolutional networks [33], and local coordina te coding [56].”
Neural Autoregressive Distribution Estimation (2016, methodology): “A NADE model for real-valued data could be obtained by applying the derivations shown in Section 2.1 to the Gaussian-RBM (Welling et al., 2005).”
Residual Energy-Based Models for Text Generation (2020, methodology): “In maximum likelihood training negatives are generated from the model, but in text application we cannot use gradient-based MCMC methods (Teh et al., 2003; Du & Mordatch, 2019) and Gibbs sampling (Welling et al., 2005) is too slow to be practical.”
Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription (2012, methodology): “Since the data consists of 49 real values per time step, we use the Gaussian RBM variant (Welling et al., 2005) for this task.”
A Kernelized Stein Discrepancy for Goodness-of-fit Tests (2016, methodology): “…mixture on which we can compare with the classical goodness-of-ﬁt tests that only work for univariate distributions, and then proceed to Gaussian-Bernoulli restricted Boltzmann machine (RBM), a graphical model widely used in deep learning (Welling et al., 2004; Hinton & Salakhutdinov, 2006).”
Deep Learning: Methods and Applications (2014, background): “See [386] for the use of general exponential-family distributions for this purpose.”
Learning the Irreducible Representations of Commutative Lie Groups (2014, neutral): “The hallmark of the deep learning approach is to learn multiple layers of representation of data, and much work has gone into the development of representation learning modules such as RBMs and their generalizations (Welling et al., 2005), and autoencoders (Vin-cent et al., 2008).”
Learning Where to Attend with Deep Architectures for Image Tracking (2011, neutral): “For the ﬁrst layers, we use (factored)-RBMs (Hinton & Salakhutdinov, 2006; Ranzato & Hinton, 2010; Welling et al., 2005; Swersky et al., 2011), but autoencoders (Vincent et al., 2008), sparse coding (Olshausen & Field, 1996; Kavukcuoglu et al., 2009), two-layer ICA (K¨oster & Hyv¨arinen, 2007) and…”
Learning Stochastic Recurrent Networks (2014, background): “E.g. RBMs are restricted to model data from the exponential family [15].”
Deep Exponential Families (2014, background): “A certain inﬁnite DEF with tied weights is equivalent to an EFH [15], but as our weights are not tied, deep exponential families represent a broader class of models than exponential family harmoniums (and RBMs).”
Real or Fake? Learning to Discriminate Machine from Human Generated Text (2019, methodology): “et al., 2006). Different learning strategies mainly differ in how negatives are mined (Ranzato et al., 2007). Some ﬁnd negatives by gradient descent, or using Monte Carlo methods like Gibbs sampling (Welling et al., 2005) and hybrid Monte Carlo (Teh et al., 2003), which enable the loss to approximat”
A tutorial survey of architectures, algorithms, and applications for deep learning (2014, background): “See [157] for the use of general exponential-family distributions for this purpose.”
Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure (2007, background): “[19] introduced a class of two-layer undirected graphical models that generalize Restricted Boltz-mann Machines (RBM’s) to exponential family distributions.”
Analyzing Drum Patterns Using Conditional Deep Belief Networks (2012, background): “There are many options for getting real-valued visible activations out of RBMs; in fact, it has been shown that every distribution in the exponential family is a viable candidate [13].”
Discovering Binary Codes for Documents by Learning Deep Generative Models (2011, background): “Welling, Rosen-Zvi, and Hinton (2005) point out that for information retrieval, fast inference is vital and to achieve this they introduce a class of two-layer undirected graphical models that generalize restricted Boltzmann machines (RBMs; see Section 2) to exponential family distributions, thus…”
Ordinal Boltzmann Machines for Collaborative Filtering (2009, result): “…explored in this paper are more general that the original proposal in (Ackley et al. , 1985) due to the use of general exponential family instead of binary variables, in the same way that the Harmoniums (Welling et al. , 2005) generalises the Restricted BMs (e.g. see (Salakhutdinov et al. , 2007)).”
Learning a Generative Model of Images by Factoring Appearance and Shape (2011, methodology): “The general foundations for using RBMs to model distributions in the exponential family were laid in Welling, Rosen-Zvi, and Hinton ( 2005 ), where one particular instantiation of this family was investigated for modeling discrete data using continuous latent variables.”