Deep Convolutional Networks as shallow Gaussian Processes

By Adrià Garriga-Alonso, Laurence Aitchison, Carl E. Rasmussen. 2018

286 citations recorded by the source corpus.

We show that the output of a (residual) convolutional neural network (CNN) with an appropriate prior over the weights and biases is a Gaussian process (GP) in the limit of infinitely many convolutional filters, extending similar results for dense networks. For a CNN, the equivalent kernel can be computed exactly and, unlike "deep kernels", has very few parameters: only the hyperparameters of the original CNN. Further, we show that this kernel has two properties that allow it to be computed efficiently; the cost of evaluating the kernel for a pair of images is similar to a single forward pass through the original CNN with only one filter per layer. The kernel equivalent to a 32-layer ResNet obtains 0.84% classification error on MNIST, a new record for GPs with a comparable number of parameters.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

In Statistical Learning Theory

Citations in the corpus

When do neural networks outperform kernel methods? (2020, background): “Some empirical studies on various datasets showed that networks can be replaced by suitable kernels with limited drop in performances [5, 21, 20, 24, 19, 10, 14, 29].”
Approximate Inference Turns Deep Networks into Gaussian Processes (2019, background): “Subsequently, many works derived explicit expressions for the GP kernel corresponding to neural networks [4, 7, 16] and their deep variants [5, 6, 13, 18].”
Similarity of Neural Network Representations Revisited (2019, methodology): “Our framework extends to any valid kernel, including kernels equivalent to neural networks (Lee et al., 2018; Jacot et al., 2018; Garriga-Alonso et al., 2019; Novak et al., 2019).”
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation (2019, methodology): “In particular, Lee et al. (2018); Garriga-Alonso et al. (2018); Novak et al. (2018) directly converted DNN to GP using this correspondence.”
A Mean Field Theory of Batch Normalization (2019, background): “Finally, we note that there is a related vein of research that has emerged that leverages the prior over functions induced by random networks to perform exact Bayesian inference (Lee et al., 2017; de G. Matthews et al., 2018; Novak et al., 2019; Garriga-Alonso et al., 2019; Yang, 2019).”
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes (2018, neutral): “In concurrent work, Garriga-Alonso et al. (2018) derive an NN-GP kernel equivalent to one of the kernels considered in our work.”
Enhanced Convolutional Neural Tangent Kernels (2019, background): “More recently, this was extended to deep and convolutional neural networks (Lee et al., 2018; Matthews et al., 2018; Novak et al., 2019; Garriga-Alonso et al., 2019).”
Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes (2019, background): “[18] Adrià Garriga-Alonso, Laurence Aitchison, and Carl Edward Rasmussen.”
Appendix for: How Good is the Bayes Posterior in Deep Neural Networks Really? (2020, unknown): “Recently a number of works added to Neal’s analysis by extending the results to deep and wide neural networks (Lee et al., 2018; de G. Matthews et al., 2018; Yang, 2019), convolutional networks (Garriga-Alonso et al., 2019), and Bayesian neural networks (Novak et al., 2019).”
Asymptotics of Wide Networks from Feynman Diagrams (2019, background): “For additional works on wide networks, including relating them to Gaussian processes, see [9–22].”
On the infinite width limit of neural networks with a standard parameterization (2020, background): “The analytic forms for the various kernels inspire some additional interesting observations: • For the NTK parameterization, the kernel resulting from a Bayesian neural network and from gradient descent training of the readout layer of an inﬁnite width network are the same.”
Dataset Meta-Learning from Kernel Ridge-Regression (2020, methodology): “Neural network kernels: Our work is motivated by the exact correspondence between infinitelywide neural networks and kernel methods (Neal, 1994; Lee et al., 2018; Matthews et al., 2018; Jacot et al., 2018; Novak et al., 2019; Garriga-Alonso et al., 2019; Arora et al., 2019a).”
Neural Operator: Learning Maps Between Function Spaces (2021, methodology): “…is to take the infinite-width limit which yields a nonparametric method and has connections to Gaussian Process Regression (Neal, 1996; Matthews et al., 2018; Garriga-Alonso et al., 2018), leading to the introduction of deep Gaussian processes (Damianou and Lawrence, 2013; Dunlop et al., 2018).”
Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel (2018, background): “Under an appropriate choice of prior, they show an equivalence between the random neural net and Gaussian processes in the limit of inﬁnite width or channels [47, 70, 36, 42, 24, 56].”
A Fine-Grained Spectral Perspective on Neural Networks (2019, background): “This connection was exploited implicitly or explicitly to build new models [15, 32, 16, 61, 62, 12, 58, 31, 10, 11, 20, 39, 34].”