Neural Tangent Kernel: Convergence and Generalization in Neural Networks

By Arthur Jacot, Franck Gabriel, Clément Hongler. 2018

4,156 citations recorded by the source corpus.

At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function (which maps input vectors to output vectors) follows the so-called kernel gradient associated with a new object, which we call the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK. We then focus on the setting of least-squares regression and show that in the infinite-width limit, the network function follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping. Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinite-width limit.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yoshua Bengio
  2. Sergey Levine
  3. Quoc V. Le
  4. Mohit Bansal
  5. Ruslan Salakhutdinov
  6. Percy Liang
  7. Jianfeng Gao
  8. Luke S. Zettlemoyer

In Statistical and Nonlinear Physics

  1. Greg Yang
  2. Yoshua Bengio
  3. Sam Schoenholz
  4. Yuanzhi Li
  5. Jeffrey Pennington
  6. Jascha Narain Sohl-Dickstein
  7. Ruslan Salakhutdinov
  8. Jason D. Lee

In Model Architectures

  1. Greg Yang
  2. Yoshua Bengio
  3. Ruslan Salakhutdinov
  4. Quoc V. Le
  5. Sam Schoenholz
  6. Sergey Levine
  7. Kyunghyun Cho
  8. Yuanzhi Li

In Scaling & Learning Theory

  1. Greg Yang
  2. Yuanzhi Li
  3. Tengyu Ma
  4. Jason D. Lee
  5. Quanquan Gu
  6. Zeyuan Allen-Zhu
  7. Simon Shaolei Du
  8. Samet Oymak

Citations in the corpus