Wide neural networks of any depth evolve as linear models under gradient descent

By Jaehoon Lee, Lechao Xiao, Sam Schoenholz, Yasaman Bahri, Roman Novak, Jascha Narain Sohl-Dickstein, Jeffrey Pennington. 2019

1,337 citations recorded by the source corpus.

A longstanding goal in deep learning research has been to precisely characterize training and generalization. However, the often complex loss landscapes of neural networks (NNs) have made a theory of learning dynamics elusive. In this work, we show that for wide NNs the learning dynamics simplify considerably and that, in the infinite width limit, they are governed by a linear model obtained from the first-order Taylor expansion of the network around its initial parameters. Furthermore, mirroring the correspondence between wide Bayesian NNs and Gaussian processes (GPs), gradient-based training of wide NNs with a squared loss produces test set predictions drawn from a GP with a particular compositional kernel. While these theoretical results are only exact in the infinite width limit, we nevertheless find excellent empirical agreement between the predictions of the original network and those of the linearized version even for finite practically-sized networks. This agreement is robust across different architectures, optimization methods, and loss functions.

Subfields: Artificial Intelligence, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yoshua Bengio
  2. Ruslan Salakhutdinov
  3. Sergey Levine
  4. Quoc V. Le
  5. Percy Liang
  6. Mohit Bansal
  7. Jianfeng Gao
  8. Dacheng Tao

In Scaling & Learning Theory

  1. Greg Yang
  2. Yuanzhi Li
  3. Tengyu Ma
  4. Jason D. Lee
  5. Zeyuan Allen-Zhu
  6. Simon Shaolei Du
  7. Colin Wei
  8. Lechao Xiao

Citations in the corpus