Unsupervised learning of distributions on binary vectors using two layer networks

By Yoav Freund, David Haussler. 1991

Subfields: Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Neural Autoregressive Distribution Estimation (2016, background): “These include the restricted Boltzmann machine (Smolensky, 1986; Freund and Haussler, 1992) and its multilayer extension, the deep Boltzmann machine (Salakhut-dinov and Hinton, 2009), which dominate the literature on undirected neural networks.”
Representation Learning: A Review and New Perspectives (2012, background): “and Haussler [66] proposed a learning algorithm for”
Learning Where to Attend with Deep Architectures for Image Tracking (2011, methodology): “We implement a multi-ﬁxation RBM very similar to the one proposed in Larochelle & Hinton (2010), where the binary variables z t (see Figure 4) are introduced to encode the relative gaze location a t within the multi-ﬁxation RBM (a “1 in K ” or “one hot” encoding of the gaze location was used for z…”
Learning a Generative Model of Images by Factoring Appearance and Shape (2011, methodology): “The base component of our model is the restricted Boltzmann machine (Smolensky, 1986 ; Freund & Haussler, 1994 ), which is a Boltzmann machine (Ackley, Hinton, & Sejnowski, 1985 ) restricted to have bipartite connectivity.”
Multimodal learning with deep Boltzmann machines (2012, background): “A Deep Boltzmann Machine (DBM) is a network of symmetrically coupled stochastic binary units.”
Modeling Human Motion Using Binary Latent Variables (2006, methodology): “We thus use a modified RBM in which the “ visible units” are linear, realvalued variables that have Gaussian noise[7, 8].”
Universal Approximation Depth and Errors of Narrow Belief Networks with Discrete Units (2013, background): “The first universal approximation result for deep and narrow sigmoid belief networks is due to Sutskever and Hinton (2008). They showed that a narrow sigmoid belief network with 3(2n − 1) + 1 layers can represent probability distributions arbitrarily close to any probability distribution on the set ”
On Contrastive Divergence Learning (2005, background): “Restricted Boltzmann machines (Smolensky, 1986; Freund and Haussler, 1992) have connections only between a hidden and a visible unit, i.e., they form a bipartite graph.”
Unsupervised Learning of Mixtures of Multiple Causes in Binary Data (1993, methodology): “…principal components analysis and its neural-network variants (Bourlard and Kamp, 1988; Sanger, 1989) as well as the Harmonium Boltzmann Machine (Freund and Haussler, 1992) are inherently multiple cause models, the hidden representations they arrive at are for many purposes intuitively…”
A Practical Guide to Training Restricted Boltzmann Machines (2012, methodology): “The training set can be modeled using a two-layer network called a “Restricted Boltzmann Machine” (Smolensky, 1986; Freund and Haussler, 1992; Hinton, 2002) in which stochastic, binary pixels are connected to stochastic, binary feature detectors using symmetrically weighted connections.”
The Shape Boltzmann Machine: A Strong Model of Object Shape (2012, background): “In fact, the hidden units can be summed out analytically (e.g. Freund and Haussler, 1994), giving rise to an alternative formulation of the RBM in terms of high-order potentials that no longer includes latent variables.”
Using deep stacking network to improve structured compressed sensing with Multiple Measurement Vectors (2013, methodology): “It uses the Restricted Boltzmann Machine (RBM) [7],[8],[9] to initialize the weights and then a Deep Stacking Network (DSN) [10] to fine tune the weights.”
Reinforcement learning for factored Markov decision processes (2002, methodology): “This case is interesting because inference and learning in this model has been intensely studied [Ackley, Hinton, and Sejnowski 1985; Hinton and Sejnowski 1986; Smolensky 1986; Freund and Haussler 1992; Hinton 2000], and the binary values are relatively easy to interpret.”
Foundations and Advances in Deep Learning (2014, methodology): “Additionally, the author proposed and tested empirically novel regularization terms for RBMs that were motivated by the contractive regularization term recently proposed for autoencoders (see Section 3.3).”
Accelerated Monte Carlo simulations with restricted Boltzmann machines (2016, background): “Given suﬃciently large number of hidden units, one can tune the parameters of an RBM to let the marginal distribution p ( x ) = h p ( x , h ) approximate any discrete distribution [48, 49].”
A Spike and Slab Restricted Boltzmann Machine (2011, background): “However before reviewing these models and their connections to the ssRBM, we note that the idea of building Boltzmann Machines with products of binary and continuous-valued variables was discussed in [26], [29], and [6].”
Classification using discriminative restricted Boltzmann machines (2008, background): “In particular, Freund and Haussler (1994); Le Roux and Bengio (2008) showed that Figure 1.”