Representational Power of Restricted Boltzmann Machines and Deep Belief Networks

By Nicolas Le Roux, Yoshua Bengio. 2008

820 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Deep AutoRegressive Networks (2013, background): “As adding extra hidden layers to models such as deep belief networks strictly improves their representational power (Le Roux & Bengio, 2008), we could ask whether that is also the case for DARN.”
Stochastic Neural Networks for Hierarchical Reinforcement Learning (2016, background): “They have rich representation power and can in fact approximate any well-behaved probability distributions (Le Roux & Bengio, 2008; Cho et al., 2013).”
PixelVAE++: Improved PixelVAE with Discrete Prior (2019, background): “Restricted Boltzmann machines model multi-modal distributions and can capture complex relationship in data [41].”
Training restricted Boltzmann machines: An introduction (2014, background): “Any distribution on {0, 1} can be modeled arbitrarily well by an RBM with m visible and k + 1 hidden units, where k denotes the cardinality of the support set of the target distribution, that is, the number of input elements from {0, 1} that have a non-zero probability of being observed [33].”
A Review of Deep Learning Models for Time Series Prediction (2019, neutral): “As such, p h ( l − 1 ) , h ( l ) refers to an RBM [168].”
Classification using discriminative restricted Boltzmann machines (2008, background): “In particular, Freund and Haussler (1994); Le Roux and Bengio (2008) showed that”
Parallel Tempering for Training of Restricted Boltzmann Machines (2010, background): “…Freund & Haussler, 1994; Hinton, 2002; Welling et al., 2005) have attracted much attention in recent years because of their power of expression (Le Roux & Bengio , 2008), because inference (of hidden variables h given visible variables v) is tractable and easy, and because they have been used…”
On the Expressive Power of Deep Architectures (2011, methodology): “With nh large enough, one can model any discrete distribution: Le Roux and Bengio (2008) showed that Restricted Boltzmann Machines (described below) are universal approximators, and since they are special cases of Boltzmann Machines, Boltzmann Machines also are universal approximators.”
Multi-Prediction Deep Boltzmann Machines (2013, methodology): “The procedures used by Le Roux and Bengio [12], Arnold and Ollivier [1] make an optimistic assumption that the deeper layers will be able to implement the best possible prior on the current layer’s hidden units.”
DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors (2018, background): “These powerful multivariate binary distributions can represent any distribution defined on a set of binary random variables [12], and have seen application in unsupervised learning [13], supervised learning [14, 15], reinforcement learning [16], dimensionality reduction [17], and collaborative filte”
Forecasting exchange rate with deep belief networks (2011, methodology): “Extensive experiments by Yoshua Bengio’s group [19] suggest that several hidden layers is better than one.”
Introduction to the BioChemical Library (BCL): An Application-Based Open-Source Toolkit for Integrated Cheminformatics and Machine Learning in Computer-Aided Drug Discovery (2022, background): “…Ma et al., 2008; Mariusz et al., 2009), Kohonen networks (KN) (Kohonen, 1990; Korolev et al., 2003; Wang et al., 2005), restricted Boltzmann machines (RBM) (Le Roux and Bengio, 2008; Tijmen Tieleman, 2008), and decision trees (DT) (Mariusz et al., 2009; Sheridan, 2012; Butkiewicz et al., 2013).”
Deep, Narrow Sigmoid Belief Networks Are Universal Approximators (2008, background): “In this paper we show that exponentially deep belief networks [3, 7, 4] can approximate any distribution over binary vectors to arbitrary accuracy, even when the width of each layer is limited to the dimensionality of the data. This resolves an open the problem in [ 6 ].”
Forecasting exchange rate using deep belief networks and conjugate gradient method (2015, methodology): “Extensive experiments by Yoshua Bengio's group [23] suggest that several hidden layers are better than one.”
Learning the Structure of Deep Sparse Graphical Models (2009, background): “While a belief network with an inﬁnitely-wide hidden layer can represent any probability distribution arbitrarily closely [Le Roux and Bengio, 2008], it is not necessarily a useful prior on such distributions.”
Joint Training Deep Boltzmann Machines for Classification (2013, methodology): “The procedures used by Le Roux and Bengio (2008); Arnold and Ollivier (2012) make an optimistic assumption that the deeper layers will be able to implement the best possible prior on the current layer’s hidden units.”
Machine Health Monitoring Using Local Feature-Based Gated Recurrent Unit Networks (2018, positive): “SAE [17] and DBNs [18] are adopted to facilitate the training of DNN and learn discriminative representation for machinery data.”