Deep Neural Networks as Gaussian Processes

By Jaehoon Lee, Yasaman Bahri, Roman Novak, Sam Schoenholz, Jeffrey Pennington, Jascha Narain Sohl-Dickstein. 2017

1,255 citations recorded by the source corpus.

It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. This correspondence enables exact Bayesian inference for infinite width neural networks on regression tasks by means of evaluating the corresponding GP. Recently, kernel functions which mimic multi-layer random neural networks have been developed, but only outside of a Bayesian framework. As such, previous work has not identified that these kernels can be used as covariance functions for GPs and allow fully Bayesian prediction with a deep neural network. In this work, we derive the exact equivalence between infinitely wide deep networks and GPs. We further develop a computationally efficient pipeline to compute the covariance function for these GPs. We then use the resulting GPs to perform Bayesian inference for wide deep neural networks on MNIST and CIFAR-10. We observe that trained neural network accuracy approaches that of the corresponding GP with increasing layer width, and that the GP uncertainty is strongly correlated with trained network prediction error. We further find that test performance increases as finite-width trained networks are made wider and more similar to a GP, and thus that GP predictions typically outperform those of finite-width networks. Finally we connect the performance of these GPs to the recent theory of signal propagation in random neural networks.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Model Architectures

In Probabilistic & Bayesian Methods

Citations in the corpus

Wide neural networks of any depth evolve as linear models under gradient descent (2019, methodology): “See [32, 33, 5, 34, 35] for more details, and [36, 7] for a formal treatment.”
The large learning rate phase of deep learning: the catapult mechanism (2020, background): “Prior work has focused on comparisons between various inﬁnite-width kernels associated with deep networks and their ﬁnite-width, SGD-trained counterparts (Lee et al., 2018; Novak et al., 2019; Arora et al., 2019).”
When do neural networks outperform kernel methods? (2020, background): “Some empirical studies on various datasets showed that networks can be replaced by suitable kernels with limited drop in performances [5, 21, 20, 24, 19, 10, 14, 29].”
On Layer Normalization in the Transformer Architecture (2020, methodology): “As the warm-up stage happens in the ﬁrst several iterations, we investigate the optimization behavior at initialization using mean ﬁeld theory (Lee et al., 2017; Xiao et al., 2018; Yang et al., 2019a; Yang, 2019; Lee et al., 2019; Zhang et al., 2019).”
Neural-net-induced Gaussian process regression for function approximation and PDE solution (2018, methodology): “However, no training was performed in [4].”
Coresets via Bilevel Optimization for Continual Learning and Streaming (2020, background): “Based on the equivalence between inﬁnitely wide networks and Gaussian processes, another option is the NNGP kernel [35].”
Approximate Inference Turns Deep Networks into Gaussian Processes (2019, background): “Subsequently, many works derived explicit expressions for the GP kernel corresponding to neural networks [4, 7, 16] and their deep variants [5, 6, 13, 18].”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, background): “Namely, GPs can be understood as the limit of infinitely wide networks (Neal, 1994; Matthews et al., 2018; Lee et al., 2018).”
Similarity of Neural Network Representations Revisited (2019, methodology): “There is thus a practical problem with invariance to invertible linear transformation: Some neural networks, especially convolutional networks, have more neurons in some layers than there are examples the training dataset (Springenberg et al., 2015; Lee et al., 2018; Zagoruyko & Komodakis, 2016).”
On Exact Computation with an Infinitely Wide Neural Net (2019, methodology): “In particular, for ReLU activation, we have E g h (cid:13) (cid:13) (cid:13) (cid:13) i Recall from [Lee et al., 2018] that in the inﬁnite width limit, the pre-activations f ( h ) ( x ) at every hidden layer h ∈ [ L ] has all its coordinates tending to i.i.d. centered Gaussian processes of…”
Gradient Descent Finds Global Minima of Deep Neural Networks (2018, neutral): “Our analysis of the Gram matrices at random initialization is closely related to prior work on the analysis of infinite-width networks as Gaussian Processes (Raghu et al., 2016; Matthews et al., 2018; Lee et al., 2017; Schoenholz et al., 2016).”
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation (2019, methodology): “In particular, Lee et al. (2018); Garriga-Alonso et al. (2018); Novak et al. (2018) directly converted DNN to GP using this correspondence.”
On the Convergence Rate of Training Recurrent Neural Networks (2018, methodology): “There is huge literature on using the mean-ﬁeld theory to study neural networks [10, 14, 25, 30, 34, 38–40, 42, 47, 56–58].”
A Mean Field Theory of Batch Normalization (2019, background): “Finally, we note that there is a related vein of research that has emerged that leverages the prior over functions induced by random networks to perform exact Bayesian inference (Lee et al., 2017; de G. Matthews et al., 2018; Novak et al., 2019; Garriga-Alonso et al., 2019; Yang, 2019).”
Gaussian Process Behaviour in Wide Deep Neural Networks (2018, methodology): “Simultaneously with the submission of the previous version of our paper to ICLR 2018, at the same conference venue, Lee et al. (2018) released a paper that has overlap with our own.”