Deep Belief Networks Are Compact Universal Approximators

By Nicolas Le Roux, Yoshua Bengio. 2010

172 citations recorded by the source corpus.

Deep belief networks (DBN) are generative models with many layers of hidden causal variables, recently introduced by Hinton, Osindero, and Teh (2006), along with a greedy layer-wise unsupervised learning algorithm. Building on Le Roux and Bengio (2008) and Sutskever and Hinton (2008), we show that deep but narrow generative networks do not require more parameters than shallow ones to achieve universal approximation. Exploiting the proof technique, we prove that deep but narrow feedforward neural networks with sigmoidal units can represent any Boolean expression.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Probabilistic & Bayesian Methods

Citations in the corpus

On the Number of Linear Regions of Deep Neural Networks (2014, background): “Other works have investigated universal approximation of probability distributions by deep belief networks (Le Roux and Bengio 2010, Mont´ufar and Ay 2011), as well as their approximation properties (Mont´ufar 2014, Krause et al. 2013).”
How to Construct Deep Recurrent Neural Networks (2013, neutral): “A number of recent theoretical results support this hypothesis (see, e.g., Le Roux and Bengio, 2010; Delalleau and Bengio, 2011; Pascanu et al. , 2013b).”
A deep learning framework for financial time series using stacked autoencoders and long-short term memory (2017, background): “According to recent studies [57, 58], better approximation to nonlinear functions can be generated by deep learning models than those models with a shallow structure.”
Hessian-based Analysis of Large Batch Training and Robustness to Adversaries (2018, methodology): “One method to defend against such adversarial attacks, is to perform robust training [30, 22]: min ✓ max x2U J (✓,x+ x, y).”
Stock prediction using deep learning (2016, background): “According to some recent papers DNNs can give better approximation to nonlinear functions than shallow models [15, 26].”
Deep Feature Extraction and Classification of Hyperspectral Images Based on Convolutional Neural Networks (2016, methodology): “There are a few deep learning models in the literature, including DBN [34], [35], SAE [36], and CNN [28].”
Learning Algorithms for the Classification Restricted Boltzmann Machine (2012, background): “Moreover, just like neural networks, ClassRBMs trained this way are universal approximators for distributionsp(y|x) with binary inputs, since RBMs are universal approximators of distributions over binary inputs (Le Roux and Bengio, 2010).”
Universal Approximation Depth and Errors of Narrow Belief Networks with Discrete Units (2013, methodology): “In the case of binary units, sequences of probability sharing steps can be defined conveniently using Gray codes, as done in (Le Roux and Bengio, 2010).”
Training restricted Boltzmann machines: An introduction (2014, methodology): “It can replace Gibbs sampling in RBM learning algorithms and is related to the Metropolized Gibbs sampler often used to sample from Ising models (Neal, 1993; Liu, 1996). Chapter 8 analyzes centered RBMs, where centering corresponds to subtracting offset values from the variables. This work is relate”
Optimal Deep Learning LSTM Model for Electric Load Forecasting using Feature Selection and Genetic Algorithm: Comparison with Machine Learning Approaches † (2018, background): “Deﬁnitely, we share the opinion that optimally trained deep learning model derives more accurate forecasting outputs than those obtained with a shallow structure [36].”
Complex-Valued Autoencoders (2011, background): “The landscape has no local minima, a family of global minima associated with Principal Component Analysis, and many families of saddle points associated with orthogonal projections onto sub-space spanned by sub-optimal subsets of eigenvectors of the covariance matrix.”
A survey of randomized algorithms for training neural networks (2016, background): “In [42], the authors showed that deep but narrow networks trained with a greedy layer-wise unsupervised learning algorithm do not require more parameters than shallow ones to achieve universal approximation.”