Wide neural networks of any depth evolve as linear models under gradient descent
By Jaehoon Lee, Lechao Xiao, Sam Schoenholz, Yasaman Bahri, Roman Novak, Jascha Narain Sohl-Dickstein, Jeffrey Pennington. 2019
1,337 citations recorded by the source corpus.
A longstanding goal in deep learning research has been to precisely characterize training and generalization. However, the often complex loss landscapes of neural networks (NNs) have made a theory of learning dynamics elusive. In this work, we show that for wide NNs the learning dynamics simplify considerably and that, in the infinite width limit, they are governed by a linear model obtained from the first-order Taylor expansion of the network around its initial parameters. Furthermore, mirroring the correspondence between wide Bayesian NNs and Gaussian processes (GPs), gradient-based training of wide NNs with a squared loss produces test set predictions drawn from a GP with a particular compositional kernel. While these theoretical results are only exact in the infinite width limit, we nevertheless find excellent empirical agreement between the predictions of the original network and those of the linearized version even for finite practically-sized networks. This agreement is robust across different architectures, optimization methods, and loss functions.
Subfields: Artificial Intelligence, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Ruslan Salakhutdinov
- Sergey Levine
- Quoc V. Le
- Percy Liang
- Mohit Bansal
- Jianfeng Gao
- Dacheng Tao
In Scaling & Learning Theory
Citations in the corpus
- Scaling Laws for Neural Language Models (2020, background): “Expansions in the model size, particularly at large width [JGH18, LXS + 19], may provide a useful framework for thinking about some of our scaling relations.”
- High-dimensional dynamics of generalization error in neural networks (2017, background): “In the lazy regime correponding to relatively large initializations, the dynamics of deep onlinear networks evolve as a linear system with features deterined by the Neural Tangent Kernel (Arora, Du, Hu, Li, & Wang, 019; Jacot, Gabriel, & Hongler, 2018; Lee et al., 2019), showing hat the frozen subsp”
- The large learning rate phase of deep learning: the catapult mechanism (2020, methodology): “For a discussion of trainability and the connection to the NTK in the lazy phase see Xiao et al. (2019).”
- When do neural networks outperform kernel methods? (2020, background): “Some empirical studies on various datasets showed that networks can be replaced by suitable kernels with limited drop in performances [5, 21, 20, 24, 19, 10, 14, 29].”
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (2020, methodology): “[16] and extended by others [2, 3, 20] shows that when the width of the layers in f tends to infinity and the learning rate for SGD tends to zero, the function f ( x ; θ ) converges over the course of training to the kernel regression solution using the neural tangent kernel (NTK), defined as: When…”
- On Layer Normalization in the Transformer Architecture (2020, methodology): “As the warm-up stage happens in the first several iterations, we investigate the optimization behavior at initialization using mean field theory (Lee et al., 2017; Xiao et al., 2018; Yang et al., 2019a; Yang, 2019; Lee et al., 2019; Zhang et al., 2019).”
- DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning (2022, background): “Recent works regarding the training dynamics of deep neural networks [16,21] have found that randomly initialized neural networks behave linearly throughout gradient descent training in the infinite-width regime.”
- One-Pass Learning via Bridging Orthogonal Gradient Descent and Recursive Least-Squares (2022, positive): “In this section, we extend our discussion to nonlinear models under the neural tangent kernel (NTK) regime [8], [10].”
- Orthogonal representations for robust context-dependent task performance in brains and neural networks (2022, positive): “low-dimensional representational patterns in the network hidden units (Chizat et al., 2018; Jacot et al., 2018; Arora et al., 2019; Lee et al., 2019;Woodworth et al., 2020).”
- Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting (2021, methodology): “The trained network is equivalent to a linearized model around random initialization (Lee et al., 2019), that is, (3) While over-parameterized models have many global minima, the NTK dynamics implicitly select the above θ , which corresponds to the L2 min-norm solution.”
- Wide Neural Networks Forget Less Catastrophically (2021, background): “, 2020), we study how the width of the network affects the continual learning performance – aligned with the recent research on infinite width networks and the neural tangent kernel (NTK) regime (Jacot et al., 2018; Lee et al., 2019; Arora et al., 2019; Chizat et al., 2019).”
- Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent (2020, background): “Lee et al. (2019) also showed that a network evolves as a linear model in the infinite width limit when trained on certain losses under gradient descent.”
- Approximate Inference Turns Deep Networks into Gaussian Processes (2019, background): “[14] derive the mean and covariance function corresponding to the GP induced by the NTK.”
- Towards Characterizing Divergence in Deep Q-Learning (2019, background): “…work shows that the NTK converges to a constant (independent of network parameters) in the limit of wide networks (Jacot et al., 2018); this result makes it possible to study the evolution of neural network functions through their linearization around the starting point (Lee et al., 2019).”