Dynamics of Deep Neural Networks and Neural Tangent Hierarchy

By Jiaoyang Huang, Horng-Tzer Yau. 2019

170 citations recorded by the source corpus.

The evolution of a deep neural network trained by the gradient descent can be described by its neural tangent kernel (NTK) as introduced in [20], where it was proven that in the infinite width limit the NTK converges to an explicit limiting kernel and it stays constant during training. The NTK was also implicit in some other recent papers [6,13,14]. In the overparametrization regime, a fully-trained deep neural network is indeed equivalent to the kernel regression predictor using the limiting NTK. And the gradient descent achieves zero training loss for a deep overparameterized neural network. However, it was observed in [5] that there is a performance gap between the kernel regression using the limiting NTK and the deep neural networks. This performance gap is likely to originate from the change of the NTK along training due to the finite width effect. The change of the NTK along the training is central to describe the generalization features of deep neural networks. In the current paper, we study the dynamic of the NTK for finite width deep fully-connected neural networks. We derive an infinite hierarchy of ordinary differential equations, the neural tangent hierarchy (NTH) which captures the gradient descent dynamic of the deep neural network. Moreover, under certain conditions on the neural network width and the data set dimension, we prove that the truncated hierarchy of NTH approximates the dynamic of the NTK up to arbitrary precision. This description makes it possible to directly study the change of the NTK for deep neural networks, and sheds light on the observation that deep neural networks outperform kernel regressions using the corresponding limiting NTK.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Model Architectures

In Scaling & Learning Theory

Citations in the corpus

The large learning rate phase of deep learning: the catapult mechanism (2020, background): “One may have hoped that including the perturbative ﬁnite-width corrections discussed in Dyer & Gur-Ari (2020); Huang & Yau (2019) would allow us to regain analytic control over the dynamics.”
Feature Learning in Infinite-Width Neural Networks (2020, background): “Training Time Many prior works (e.g. (Allen-Zhu et al., 2018; Huang & Yau, 2019; Mei et al., 2018)) derived explicit time dependence of the convergence to infinite-width limit, so that a larger width can allow the network to stay close to the limit for longer.”
Mean-field Behaviour of Neural Tangent Kernel for Deep Neural Networks (2019, background): “For ReLU, a similar result appeared independently in (Huang et al., 2020) after the ﬁrst version of this paper was made publicly available.”
TRAK: Attributing Model Behavior at Scale (2023, unknown): “For example, incorporating higher order approximations (Huang & Yau, 2020; Bai & Lee, 2020) beyond the linear approximation used in TRAK is a possible direction.”
Asymptotics of Wide Networks from Feynman Diagrams (2019, background): “The Neural Tangent Hierarchy presented in [26], published during the completion of this version, has significant overlap with the recursive differential equations (11) presented below.”
Dataset Distillation with Infinitely Wide Convolutional Networks (2021, methodology): “…1994, Lee et al., 2018, Matthews et al., 2018, Jacot et al., 2018, Novak et al., 2019, Garriga-Alonso et al., 2019, Arora et al., 2019, Yang, 2019a,b, Hron et al., 2020] and the extended correspondence with the finite width corrections [Dyer and Gur-Ari, 2020, Huang and Yau, 2020, Yaida, 2020].”
Disentangling trainability and generalization in deep learning (2019, background): “Since ˆΘ converges to Θ , the gradient ﬂow dynamics of real network also converge to the dynamics described by Equation 4 and Equation 5 (Jacot et al., 2018; Lee et al., 2019; Chizat et al., 2019; Yang, 2019; Arora et al., 2019; Huang & Yau, 2019).”
Frequency Bias in Neural Networks for Input of Non-Uniform Density (2020, background): “…applied to two-layer networks (Bach, 2017; Bietti & Mairal, 2019; Du et al., 2019; Vempala & Wilmes, 2018; Xie et al., 2017), multilayer perceptrons (i,e. fully connected), residual and convolutional networks (Allen-Zhu et al., 2018; 2019; Arora et al., 2019a; Huang & Yau, 2019; Lee et al., 2019).”
Backward Feature Correction: How Deep Learning Performs Deep Learning (2020, background): “For such reason, it is not surprising that most existing theoretical works on the efficient learning regime of neural networks either focus on (1) two-layer networks [9, 10, 12, 22, 23, 36, 41, 41, 42, 46, 50, 51, 53, 54, 56, 58, 61] which do not have any hierarchical structure, or (2) a multi-layer”