Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes

By Greg Yang. 2019

243 citations recorded by the source corpus.

Wide neural networks with random weights and biases are Gaussian processes, as observed by Neal (1995) for shallow networks, and more recently by Lee et al.~(2018) and Matthews et al.~(2018) for deep fully-connected networks, as well as by Novak et al.~(2019) and Garriga-Alonso et al.~(2019) for deep convolutional networks. We show that this Neural Network-Gaussian Process correspondence surprisingly extends to all modern feedforward or recurrent neural networks composed of multilayer perceptron, RNNs (e.g. LSTMs, GRUs), (nD or graph) convolution, pooling, skip connection, attention, batch normalization, and/or layer normalization. More generally, we introduce a language for expressing neural network computations, and our result encompasses all such expressible neural networks. This work serves as a tutorial on the \emph{tensor programs} technique formulated in Yang (2019) and elucidates the Gaussian Process results obtained there. We provide open-source implementations of the Gaussian Process kernels of simple RNN, GRU, transformer, and batchnorm+ReLU network at github.com/thegregyang/GP4A. Please see our arxiv version for the complete and up-to-date version of this paper.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

In Statistical Learning Theory

Citations in the corpus

Feature Learning in Infinite-Width Neural Networks (2020, background): “5556 55It is completely OK to define an initial scalar using randomness from other parts of the program, as long as this scalar converges almost surely to a deterministic limit 56We cannot define it using a Moment instruction because, intuitively, the mechanism of this convergence is through CLT, no”
Dataset Distillation with Infinitely Wide Convolutional Networks (2021, background): “…[Neal, 1994, Lee et al., 2018, Matthews et al., 2018, Jacot et al., 2018, Novak et al., 2019, Garriga-Alonso et al., 2019, Arora et al., 2019, Yang, 2019a,b, Hron et al., 2020] and the extended correspondence with the finite width corrections [Dyer and Gur-Ari, 2020, Huang and Yau, 2020,…”
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (2022, background): “This is because the neural network in μP is approximately a Gaussian process (GP) at initialization with variance on the order Θ(σ(2)/width) (contrast this with SP networks, which approximates a GP with Θ(σ(2)) variance) [50, 54].”
Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics (2021, methodology): “(Yang, 2019a; 2020a) showed that any neural network of standard architecture is represented by a N ETSOR + program.”
Tensor Programs II: Neural Tangent Kernel for Any Architecture (2020, methodology): “Nonlin Like NETSOR in Yang [62] for forward propagation, NETSOR> can express all standard architectures.”
Tensor Programs III: Neural Matrix Laws (2020, methodology): “As with the NNGP in Appendix B, this infinite-width NTK can be computed in a straightforward way using Theorem 2.10, following the examples of Yang [51].”
On the infinite width limit of neural networks with a standard parameterization (2020, background): “The analytic forms for the various kernels inspire some additional interesting observations: • For the NTK parameterization, the kernel resulting from a Bayesian neural network and from gradient descent training of the readout layer of an inﬁnite width network are the same.”