Finite Depth and Width Corrections to the Neural Tangent Kernel

By Boris Hanin, Mihai Nica. 2019

175 citations recorded by the source corpus.

We prove the precise scaling, at finite depth and width, for the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network. The standard deviation is exponential in the ratio of network depth to width. Thus, even in the limit of infinite overparameterization, the NTK is not deterministic if depth and width simultaneously tend to infinity. Moreover, we prove that for such deep and wide networks, the NTK has a non-trivial evolution during training by showing that the mean of its first SGD update is also exponential in the ratio of network depth to width. This is sharp contrast to the regime where depth is fixed and network width is very large. Our results suggest that, unlike relatively shallow and wide networks, deep and wide ReLU networks are capable of learning data-dependent features even in the so-called lazy training regime.

Subfields: Artificial Intelligence, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Scaling & Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, methodology): “Finally, our analysis is based on a scaling estimate of the fluctuations of the NTK at initialization, recently supported by more detailed analysis based on Feynman diagrams and path numbering [40, 41].”
Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK (2020, unknown): “…and Gu (2019); Du et al. (2018c); Arora et al. (2019a); Allen-Zhu and Li (2019b); Allen-Zhu et al. (2019c,b); Li and Liang (2018); Zou et al. (2018); Du et al. (2018a); Daniely et al. (2016); Ghorbani et al. (2019); Li et al. (2019a); Hanin and Nica (2019); Yang (2019) and the references therein.”
Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning (2020, background): “, ensemble) in deep learning works very differently from model averaging in random feature mappings — in particular, different from the neural tangent kernel (NTK) approach [3, 6, 7, 9, 10, 20, 25, 27, 28, 40, 41, 45, 55, 59, 86, 92].”
Disentangling trainability and generalization in deep learning (2019, background): “We point out that the degradation of the agreement for larger depth may due to the fact that the ﬁnite width NTK becomes more stochastic as the ratio between depth and width increases (Hanin & Nica, 2019).”
Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics (2021, methodology): “Finite width corrections to the NTK were derived for fully connected networks in (Hanin & Nica, 2019; Littwin et al., 2020a).”
Tensor Programs II: Neural Tangent Kernel for Any Architecture (2020, background): “Other works studied the higher order terms in the Taylor expansion [21, 36], ensembled NTK [49], or finite width corrections [27, 47].”
Mean-field Behaviour of Neural Tangent Kernel for Deep Neural Networks (2019, background): “…width neural networks changes with time (Chizat and Bach, 2018; Ghorbani et al., 2019; Huang and Yau, 2020), and might even be random as shown by (Hanin and Nica, 2019) where authors prove that in the limit n, L → ∞ (where n is a width of the network) with ﬁxed ratio γ = Ln , the limiting kernel…”
Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective (2021, background): “(Jacot et al., 2018) or ﬁnite wide deep networks (Hanin & Nica, 2019).”
Explaining neural scaling laws (2021, methodology): “In this section we consider training and test data drawn uniformly from a compact d -dimensional manifold, x ∈ M d , and targets given by some smooth function y = F ( x ) on this manifold.”