Steps Toward Deep Kernel Methods from Infinite Neural Networks

By Tamir Hazan, Tommi S. Jaakkola. 2015

85 citations recorded by the source corpus.

Contemporary deep neural networks exhibit impressive results on practical problems. These networks generalize well although their inherent capacity may extend significantly beyond the number of training examples. We analyze this behavior in the context of deep, infinite neural networks. We show that deep infinite layers are naturally aligned with Gaussian processes and kernel methods, and devise stochastic kernels that encode the information of these networks. We show that stability results apply despite the size, offering an explanation for their empirical success.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Deriving Neural Architectures from Sequence and Graph Kernels (2017, background): “Our work follows recent work demonstrating the connection between neural networks and kernels (Cho & Saul, 2009; Hazan & Jaakkola, 2015).”
Deep Neural Networks as Gaussian Processes (2017, background): “There is related work on compositional kernels and the underlying recurrence relation and can be found in ([2, 7, 4, 13, 15]).”
On Exact Computation with an Infinitely Wide Neural Net (2019, background): “Follow-up work extended this correspondence to more general shallow neural networks [Williams, 1997, Roux and Bengio, 2007, Hazan and Jaakkola, 2015].”
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation (2019, background): “Later works extended the conditions under which this scaling limit takes place (Williams, 1997; Le Roux & Bengio, 2007; Hazan & Jaakkola, 2015).”
Gaussian Process Behaviour in Wide Deep Neural Networks (2018, methodology): “Next we state the following lemma, which we attribute to Hazan and Jaakkola (2015): Lemma 2 (Normal recursion) If the activations of a previous layer are normally distributed with moments: Then under the recursion (2) and as H → ∞ the activations of the next layer converge in distribution to a…”
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes (2018, background): “Follow-up work extended the conditions under which this correspondence applied [41, 22, 17].”
Variational Implicit Processes (2018, background): “[68,69,91] for single-hidden layer, and [27,33,53,62] for deeper nets.”
Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics (2021, background): “…view of wide neural networks, which characterizes the behaviour of training only the last layer of a wide neural network, has been studied in (Daniely et al., 2016; Hazan & Jaakkola, 2015; Roux & Ben-gio, 2007; Lee et al., 2018; Matthews et al., 2018; Hinton & Neal, 1995; Novak et al., 2019).”
Tensor Programs III: Neural Matrix Laws (2020, background): “This Neural Network-Gaussian Process correspondence has become a fascinating topic ever since, with a flurry of activities in recent years extending it [9, 17, 22, 23, 28, 32, 47].”
A Fine-Grained Spectral Perspective on Neural Networks (2019, background): “The Gaussian process behavior of neural networks was found by Neal [37] for shallow networks and then extended over the years to different settings and architectures [59, 32, 25, 16, 33, 36, 38].”
Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks (2019, methodology): “The connection between inﬁnitely wide neural networks and kernel methods is not new (Neal, 1996; Williams, 1997; Roux & Bengio, 2007; Hazan & Jaakkola, 2015; Lee et al., 2018; Matthews et al., 2018; Novak et al., 2019; Garriga-Alonso et al., 2019; Cho & Saul, 2009; Daniely et al., 2016; Daniely,…”
Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes (2019, background): “This result was extended over time [56, 34, 22, 13], and finally to deep, fully-connected or convolutional, neural networks in recent years [37, 40, 43].”
Nonparametric Neural Networks (2017, methodology): “Maclaurin et al. (2015) indeed calculate the gradient of the validation error after training with respect to certain hyperparameters, though their method only applies to specific networks trained with very specific algorithms.”