Gaussian Process Behaviour in Wide Deep Neural Networks

By Alexander G. de G. Matthews, Mark Rowland, Jiri Hron, Richard E. Turner, Zoubin Ghahramani. 2018

619 citations recorded by the source corpus.

Whilst deep neural networks have shown great empirical success, there is still much work to be done to understand their theoretical properties. In this paper, we study the relationship between random, wide, fully connected, feedforward networks with more than one hidden layer and Gaussian processes with a recursive kernel definition. We show that, under broad conditions, as we make the architecture increasingly wide, the implied random function converges in distribution to a Gaussian process, formalising and extending existing results by Neal (1996) to deep networks. To evaluate convergence rates empirically, we use maximum mean discrepancy. We then compare finite Bayesian deep networks from the literature to Gaussian processes in terms of the key predictive quantities of interest, finding that in some cases the agreement can be very close. We discuss the desirability of Gaussian process behaviour and review non-Gaussian alternative models from the literature.

Subfields: Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

Citations in the corpus

Wide neural networks of any depth evolve as linear models under gradient descent (2019, methodology): “Therefore, randomly initialized neural networks are in correspondence with a certain class of GPs (hereinafter referred to as NNGPs), which facilitates a fully Bayesian treatment of neural networks [5, 6].”
Neural Tangent Kernel: Convergence and Generalization in Neural Networks (2018, background): “In the inﬁnite-width limit, ANNs have a Gaussian distribution described by a kernel (16; 4; 7; 13; 6).”
When do neural networks outperform kernel methods? (2020, background): “Some empirical studies on various datasets showed that networks can be replaced by suitable kernels with limited drop in performances [5, 21, 20, 24, 19, 10, 14, 29].”
Approximate Inference Turns Deep Networks into Gaussian Processes (2019, background): “Subsequently, many works derived explicit expressions for the GP kernel corresponding to neural networks [4, 7, 16] and their deep variants [5, 6, 13, 18].”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, background): “Namely, GPs can be understood as the limit of infinitely wide networks (Neal, 1994; Matthews et al., 2018; Lee et al., 2018).”
Deep Neural Networks as Gaussian Processes (2017, background): “A concurrent work (Anonymous, 2018) further derives the convergence rate towards a GP if all layers are taken to infinite width simultaneously.”
On Exact Computation with an Infinitely Wide Neural Net (2019, methodology): “More recently, this was extended to deep and convolutional neural networks [Lee et al., 2018, Matthews et al., 2018, Novak et al., 2019, Garriga-Alonso et al., 2019].”
Gradient Descent Finds Global Minima of Deep Neural Networks (2018, neutral): “Our analysis of the Gram matrices at random initialization is closely related to prior work on the analysis of infinite-width networks as Gaussian Processes (Raghu et al., 2016; Matthews et al., 2018; Lee et al., 2017; Schoenholz et al., 2016).”
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation (2019, background): “Recently, Lee et al. (2018); Matthews et al. (2018) empirically and/or theoretically investigated analogous correspondences for infinite width, finite depth deep MLPs, and likewise Novak et al. (2018), for deep convolution networks.”
Neural Tangents: Fast and Easy Infinite Neural Networks in Python (2019, methodology): “Since then, this correspondence has been expanded to a wide range of nonlinearities (Matthews et al., 2018b; Novak et al., 2019) and ∗Equal contribution.”
A Mean Field Theory of Batch Normalization (2019, background): “Finally, we note that there is a related vein of research that has emerged that leverages the prior over functions induced by random networks to perform exact Bayesian inference (Lee et al., 2017; de G. Matthews et al., 2018; Novak et al., 2019; Garriga-Alonso et al., 2019; Yang, 2019).”
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes (2018, methodology): “We prove convergence as the number of channels in hidden layers approach infinity simultaneously (i.e. min { n1, . . . , nL } →∞, see §E.4 for details), extending the result of Matthews et al. (2018a) under mild conditions on the nonlinearity derivative.”
Enhanced Convolutional Neural Tangent Kernels (2019, background): “More recently, this was extended to deep and convolutional neural networks (Lee et al., 2018; Matthews et al., 2018; Novak et al., 2019; Garriga-Alonso et al., 2019).”
Variational Bayesian dropout: pitfalls and fixes (2018, background): “For a subset of architectures and priors, Item 1 can be addressed by studying limit behaviour of increasingly large networks (see, for example, (Neal, 1996; Matthews et al., 2018)); in other cases, sensibility of P( W ) must be assessed individually.”
On the Selection of Initialization and Activation Function for Deep Neural Networks (2018, methodology): “) by a Gaussian process was ﬁrst proposed by Neal (1995) in the single layer case and has been recently extended to the multiple layer case by Lee et al. (2018) and Matthews et al. (2018).”
A Fine-Grained Spectral Perspective on Neural Networks (2019, background): “The Gaussian process behavior of neural networks was found by Neal [37] for shallow networks and then extended over the years to different settings and architectures [59, 32, 25, 16, 33, 36, 38].”