On Exact Computation with an Infinitely Wide Neural Net

By Sanjeev Arora, Simon Shaolei Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, Ruosong Wang. 2019

1,059 citations recorded by the source corpus.

How well does a classic deep net architecture like AlexNet or VGG19 classify on a standard dataset such as CIFAR-10 when its width --- namely, number of channels in convolutional layers, and number of nodes in fully-connected internal layers --- is allowed to increase to infinity? Such questions have come to the forefront in the quest to theoretically understand deep learning and its mysteries about optimization and generalization. They also connect deep learning to notions such as Gaussian processes and kernels. A recent paper [Jacot et al., 2018] introduced the Neural Tangent Kernel (NTK) which captures the behavior of fully-connected deep nets in the infinite width limit trained by gradient descent; this object was implicit in some other recent papers. An attraction of such ideas is that a pure kernel-based method is used to capture the power of a fully-trained deep net of infinite width. The current paper gives the first efficient exact algorithm for computing the extension of NTK to convolutional neural nets, which we call Convolutional NTK (CNTK), as well as an efficient GPU implementation of this algorithm. This results in a significant new benchmark for the performance of a pure kernel-based method on CIFAR-10, being $10\%$ higher than the methods reported in [Novak et al., 2019], and only $6\%$ lower than the performance of the corresponding finite deep net architecture (once batch normalization, etc. are turned off). Theoretically, we also give the first non-asymptotic proof showing that a fully-trained sufficiently wide net is indeed equivalent to the kernel regression predictor using NTK.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yoshua Bengio
  2. Ruslan Salakhutdinov
  3. Tengyu Ma
  4. Percy Liang
  5. Sergey Levine
  6. Quoc V. Le
  7. Mohit Bansal
  8. Yuanzhi Li

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Ming-Hsuan Yang
  5. Jitendra Malik
  6. Xiaogang Wang
  7. Andrew Zisserman
  8. Andrea Vedaldi

In Large Language Models

  1. Yuanzhi Li
  2. Greg Yang
  3. Zeyuan Allen-Zhu
  4. Lechao Xiao
  5. Yingyu Liang
  6. Simon Shaolei Du
  7. Aaron C. Courville
  8. Yoshua Bengio

In Scaling & Learning Theory

  1. Greg Yang
  2. Yuanzhi Li
  3. Zeyuan Allen-Zhu
  4. Jason D. Lee
  5. Simon Shaolei Du
  6. Tengyu Ma
  7. Quanquan Gu
  8. Lechao Xiao

Citations in the corpus