Neural Tangent Kernel: Convergence and Generalization in Neural Networks
By Arthur Jacot, Franck Gabriel, Clément Hongler. 2018
4,156 citations recorded by the source corpus.
At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function (which maps input vectors to output vectors) follows the so-called kernel gradient associated with a new object, which we call the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK. We then focus on the setting of least-squares regression and show that in the infinite-width limit, the network function follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping. Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinite-width limit.
Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Quoc V. Le
- Mohit Bansal
- Ruslan Salakhutdinov
- Percy Liang
- Jianfeng Gao
- Luke S. Zettlemoyer
In Statistical and Nonlinear Physics
- Greg Yang
- Yoshua Bengio
- Sam Schoenholz
- Yuanzhi Li
- Jeffrey Pennington
- Jascha Narain Sohl-Dickstein
- Ruslan Salakhutdinov
- Jason D. Lee
In Model Architectures
- Greg Yang
- Yoshua Bengio
- Ruslan Salakhutdinov
- Quoc V. Le
- Sam Schoenholz
- Sergey Levine
- Kyunghyun Cho
- Yuanzhi Li
In Scaling & Learning Theory
- Greg Yang
- Yuanzhi Li
- Tengyu Ma
- Jason D. Lee
- Quanquan Gu
- Zeyuan Allen-Zhu
- Simon Shaolei Du
- Samet Oymak
Citations in the corpus
- Scaling Laws for Neural Language Models (2020, background): “Expansions in the model size, particularly at large width [JGH18, LXS + 19], may provide a useful framework for thinking about some of our scaling relations.”
- Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (2019, background): “Indeed, recent theoretical works [Jacot et al., 2018, Du et al., 2019, Zhang et al., 2019a] show that for wide enough networks, the weights stay close enough to the initialization for the linearized approximation to remain accurate.”
- Wide neural networks of any depth evolve as linear models under gradient descent (2019, methodology): “It has already been adopted in several recent works (van Laarhoven, 2017; Karras et al., 2018; Jacot et al., 2018; Du et al., 2018; Park et al., 2018).”
- Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, background): “, 2011; Dozat, 2016; Martens & Grosse, 2015)) has become an important design choice, though understanding of its relationship to model generalization remains nascent (Shirish Keskar et al., 2016; Wilson et al., 2017; Shirish Keskar & Socher, 2017; Agarwal et al., 2020; Jacot et al., 2018).”
- The large learning rate phase of deep learning: the catapult mechanism (2020, methodology): “…its initial parameters (Lee et al., 2019); this becomes exact in the infinite width limit, where these dynamics are sometimes called lazy training (Jacot et al., 2018; Lee et al., 2019; Du et al., 2019; Li & Liang, 2018; Zou et al., 2018; Allen-Zhu et al., 2019; Chizat et al., 2019; Dyer &…”
- The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks (2020, background): “Key to our technical analysis is a bound on the spectral norm of the difference between the Neural Tangent Kernel (NTK) [Jacot et al., 2018] of the neural network at initialization and that of the linear model; indeed, a weaker result, like a bound on the Frobenius norm, would be insufficient to…”
- Feature Purification: How Adversarial Training Performs Robust Deep Learning (2020, background): “operty of a random initialization; or (2) only consider the neural tangent kernel regime, where the neural networks are linearized so the features are not learned (they stay at random initialization) [4, 6{8, 13, 14, 26, 26{29, 38, 45, 50, 57, 61, 104, 114, 115]. 1 Cleaning Training global feature l”
- Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian (2019, background): “Definition 3.1 (Multiclass Neural Tangent Kernel (M-NTK) [31]) Let w ∈ Rd be a vector with N (0,Id) distribution.”
- A Convergence Theory for Deep Learning via Over-Parameterization (2018, background): “The neural tangent kernel (NTK) [32] is usually referred to as the feature space defined by the network gradient at random initialization.”
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (2020, methodology): “[16] and extended by others [2, 3, 20] shows that when the width of the layers in f tends to infinity and the learning rate for SGD tends to zero, the function f ( x ; θ ) converges over the course of training to the kernel regression solution using the neural tangent kernel (NTK), defined as: When…”
- Conservative Q-Learning for Offline Reinforcement Learning (2020, background): “2 to the case of both linear function approximators and non-linear neural network function approximators, where the latter builds on the neural tangent kernel (NTK) framework [25].”
- Differentially Private Learning Needs Better Features (or Much More Data) (2020, background): “Recent work on Neural Tangent Kernels (Jacot et al., 2018) has shown that the performance of deep neural networks on CIFAR-10 could be matched by specialized kernel methods (Li et al.”
- DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning (2022, background): “In [16], it is shown that such model can be described by a specific kernel function, coined neural tangent kernel, defined by the first-order derivatives of the neural network.”
- One-Pass Learning via Bridging Orthogonal Gradient Descent and Recursive Least-Squares (2022, positive): “In this section, we extend our discussion to nonlinear models under the neural tangent kernel (NTK) regime [8], [10].”
- Orthogonal representations for robust context-dependent task performance in brains and neural networks (2022, positive): “low-dimensional representational patterns in the network hidden units (Chizat et al., 2018; Jacot et al., 2018; Arora et al., 2019; Lee et al., 2019;Woodworth et al., 2020).”
- Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting (2021, methodology): “Then, the model update on the the class 1 at the n -th task, i.e., f ) in the infinite-width limit (Jacot et al., 2018; Yang, 2019).”