Deep, Narrow Sigmoid Belief Networks Are Universal Approximators

By Ilya Sutskever, Geoffrey E. Hinton. 2008

156 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Learning Summary Statistic for Approximate Bayesian Computation via Deep Neural Network (2015, background): “ network, the DNN gains the power to approximate any continuous function; however, rigorous proof of the approximation properties of DNNs remains an important open problem (Farago and Lugosi (1993); Sutskever and Hinton (2008); Le Roux and Bengio (2010)). Nonetheless we expect that DNNs can eectivel”
Optimizing Neural Networks that Generate Iimages (2014, background): “Any smooth function can be well approximated by very shallow networks of exponential width, or very narrow networks of exponential depth (Sutskever & Hinton, 2008).”
Spectral–Spatial Classification of Hyperspectral Data Based on Deep Belief Network (2015, unknown): “The process of fine-tuning is back-propagation, searching for a minimum in a peripheral region of parameters initialized by DBN [47], [48].”
Universal Approximation Depth and Errors of Narrow Belief Networks with Discrete Units (2013, methodology): “We relax the setting of binary units (Sutskever and Hinton, 2008; Le Roux and Bengio, 2008, 2010; Montúfar and Ay, 2011) to units with arbitrary finite state spaces, and the vanishing approximation error to an arbitrary approximation error tolerance.”
Refinements of Universal Approximation Results for Deep Belief Networks and Restricted Boltzmann Machines (2010, background): “An important idea of Sutskever and Hinton (2008) is that of sharing, by means of which in a part of a DBN the probability of a vector is increased while the probability of another vector is decreased and the probability of all other vectors remains nearly constant.”
Two Distributed-State Models For Generating High-Dimensional Time Series (2011, background): “Keywords: unsupervised learning, restricted Boltzmann machines, time series, generative models, motion capture”
On the number of response regions of deep feed forward networks with piece-wise linear activations (2013, methodology): “…the representational power of generative models based on Boltzmann machines Mont´ufar et al. (2011); Martens et al. (2013) and deep belief networks (Sutskever and Hinton, 2008; Le Roux and Bengio, 2010; Mont´ufar and Ay, 2011), or have compared mixtures and products of experts models (Mont´ufar…”
Stock prediction using deep learning (2016, background): “According to some recent papers DNNs can give better approximation to nonlinear functions than shallow models [15, 26].”
Universal Approximation with Deep Narrow Networks (2019, unknown): “See also Sutskever and Hinton [SH08].”
Boolean autoencoders and hypercube clustering complexity (2012, background): “In spite of the interest they have generated, and with few exceptions [3,26], little theoretical understanding of autoencoders and deep architectures has been obtained to this date, especially in the non-linear case.”
Deep Belief Networks Are Compact Universal Approximators (2010, methodology): “Sutskever and Hinton (2008) showed that a deep but narrow DBN (with only n + 1 units per layer and n binary inputs) can represent any distribution on its input, with no more than 3 × 2 n layers.”
On building ensembles of stacked denoising auto-encoding classifiers and their further improvement (2018, background): “There are also proofs of universal approximation capabilities for DNNs [20,21] , as well as of some interesting characteristics of them 1 [22] .”