Computing with Infinite Networks

By Christopher K. I. Williams. 1996

Subfields: Probabilistic & Bayesian Methods, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

In Statistical Learning Theory

Citations in the corpus

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning (2015, background): “It has long been known that infinitely wide (single hidden layer) NNs with distributions placed over their weights converge to Gaussian processes [15, 16].”
Scalable Bayesian Optimization Using Deep Neural Networks (2015, methodology): “Bayesian neural networks are a natural consideration, not least because of the theoretical relationship between Gaussian processes and inﬁ-nite Bayesian neural networks (Neal, 1995; Williams, 1996).”
Deep Neural Networks as Gaussian Processes (2017, methodology): “Williams (1997) computes analytic GP kernels for single hidden-layer neural networks with error function or Gaussian nonlinearities and noted the use of the GP prior for exact Bayesian inference in regression.”
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation (2019, background): “Later works extended the conditions under which this scaling limit takes place (Williams, 1997; Le Roux & Bengio, 2007; Hazan & Jaakkola, 2015).”
A Mean Field Theory of Batch Normalization (2019, background): “These integrals can either be performed analytically (Cho & Saul, 2009; Williams, 1997) or efficiently approximated numerically (Lee et al.”
Gaussian Process Behaviour in Wide Deep Neural Networks (2018, background): “Historically this result was significant because it provided a connection between flexible Bayesian neural networks and Gaussian processes (Williams, 1998; Rasmussen and Williams, 2006)”
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes (2018, methodology): “…affine transformation of the kernel (see Equation 5), and C is a well-studied map in literature (see §G.4), and for nonlinearities such as ReLU (Nair & Hinton, 2010) and the error function (erf) C can be computed in closed form as derived in Cho & Saul (2009) and Williams (1997) respectively.”
Leveraging uncertainty information from deep neural networks for disease detection (2016, background): “Recently, it was shown 30 that a multi-layer-perceptron (i.e. a stack of densely connected layers) with dropout after every weight layer is math-ematically equivalent to approximate variational inference 51 in the deep GP model 65,66 .”
Learning Shared Latent Structure for Image Synthesis and Robotic Imitation (2005, methodology): “Gaussian processes (GPs) are powerful models for classification and regression that subsume numerous classes of function approximators, such as single hidden-layer neural networks and RBF networks [8, 15, 9].”
Nonparametric Neural Networks (2017, methodology): “Gaussian processes can been used to mimick “inﬁnitely wide” neural networks (e.g. Williams (1997), Hazan & Jaakkola (2015)), thus eliminating the need to choose layer width and replacing it with the need to choose a kernel.”
Sparse Spectrum Gaussian Process Regression (2010, result): “This is analogous to the result of Williams (1997) for the non-stationary multilayer perceptron covariance function.”
Bayesian Classification With Gaussian Processes (1998, background): “This covariance function can be obtained from a network of Gaussian radial basis functions in the limit of an infinite number of hidden units [27].”
Practical options for selecting data-driven or physics-based prognostics algorithms with reviews (2015, background): “The global model often takes the form of constant or polynomials, while the covariance function includes radial basis (or squared exponential), rational quadratic, neural network, Matern, periodic, constant and linear functions [81,82].”
Upper and Lower Bounds on the Learning Curve for Gaussian Processes (2000, methodology): “Williams (1997) calculated the covariance functions of GPs correspondingto neural networks with certain weight priors and transfer functions.”