Refinements of Universal Approximation Results for Deep Belief Networks and Restricted Boltzmann Machines

By Guido Montúfar, Nihat Ay. 2010

96 citations recorded by the source corpus.

We improve recently published results about resources of restricted Boltzmann machines (RBM) and deep belief networks (DBN) required to make them universal approximators. We show that any distribution on the set of binary vectors of length can be arbitrarily well approximated by an RBM with hidden units, where is the minimal number of pairs of binary vectors differing in only one entry such that their union contains the support set of . In important cases this number is half the cardinality of the support set of (given in Le Roux & Bengio, 2008). We construct a DBN with , hidden layers of width that is capable of approximating any distribution on arbitrarily well. This confirms a conjecture presented in Le Roux and Bengio (2010).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

On the Number of Linear Regions of Deep Neural Networks (2014, background): “Other works have investigated universal approximation of probability distributions by deep belief networks (Le Roux and Bengio 2010, Mont´ufar and Ay 2011), as well as their approximation properties (Mont´ufar 2014, Krause et al. 2013).”
Training restricted Boltzmann machines: An introduction (2014, background): “It has been shown recently that even fewer units can be sufficient, depending on the patterns in the support set [41].”
Neural Networks and Deep Learning (2018, unknown): “The universal approximation property of RBMs is discussed in [341].”
On the number of response regions of deep feed forward networks with piece-wise linear activations (2013, background): “…power of generative models based on Boltzmann machines Mont´ufar et al. (2011); Martens et al. (2013) and deep belief networks (Sutskever and Hinton, 2008; Le Roux and Bengio, 2010; Mont´ufar and Ay, 2011), or have compared mixtures and products of experts models (Mont´ufar and Morton, 2012).”
How Can Deep Rectifier Networks Achieve Linear Separability and Preserve Distances? (2015, background): “Most of the theoretical works have focused on the universal approximation power of deep neural networks for functions (Hornik et al., 1989) or for probability distributions (Le Roux & Bengio, 2010; Montufar & Ay, 2011).”
Approximation properties of DBNs with binary hidden units and real-valued visible units (2013, unknown): “The number of hidden neurons required can be reduced to the minimum number of pairs of input vectors differing in only one component with the property that their union contains all observable patterns having positive probability (Montufar & Ay, 2011).”
Universal Approximation Depth and Errors of Narrow Belief Networks with Discrete Units (2013, methodology): “The superposition of probability sharing steps is inspired by (Le Roux and Bengio, 2010), together with the refinements of that work devised in (Montúfar and Ay, 2011).”
Training Restricted Boltzmann Machines (2015, background): “Binary RBMs, in which each variable conditioned on the others is Bernoulli distributed, are able to approximate arbitrarily well any distribution over the observable variables (Le Roux and Bengio, 2008; Montufar and Ay, 2011).”
Exploiting the deep learning paradigm for recognizing human actions (2014, background): “Several theoretical demonstrations have been provided about the optimal number of hidden units able to represent any data distributions [16][19][12].”