Universal Approximation Depth and Errors of Narrow Belief Networks with Discrete Units

By Guido Montúfar. 2013

41 citations recorded by the source corpus.

We generalize recent theoretical work on the minimal number of layers of narrow deep belief networks that can approximate any probability distribution on the states of their visible units arbitrarily well. We relax the setting of binary units (Sutskever & Hinton, 2008; Le Roux & Bengio, 2008, 2010; Montúfar & Ay, 2011) to units with arbitrary finite state spaces and the vanishing approximation error to an arbitrary approximation error tolerance. For example, we show that a q-ary deep belief network with layers of width for some can approximate any probability distribution on without exceeding a Kullback-Leibler divergence of . Our analysis covers discrete restricted Boltzmann machines and naive Bayes models as special cases.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Probabilistic & Bayesian Methods

Citations in the corpus

On the Number of Linear Regions of Deep Neural Networks (2014, background): “Other works have investigated universal approximation of probability distributions by deep belief networks (Le Roux and Bengio 2010, Mont´ufar and Ay 2011), as well as their approximation properties (Mont´ufar 2014, Krause et al. 2013).”
Nonparametric regression using deep neural networks with ReLU activation function (2017, background): “The number of required hidden units might be, however, extremely large; cf. [37] and [36].”
Neural Networks and Deep Learning (2018, unknown): “Although a detailed proof [340] of this fact is beyond the scope of this book, we provide a simple example to elucidate this point.”