On Lazy Training in Differentiable Programming

By Lénaïc Chizat, Edouard Oyallon, Francis R. Bach. 2018

1,024 citations recorded by the source corpus.

In a series of recent theoretical works, it was shown that strongly over-parameterized neural networks trained with gradient-based methods could converge exponentially fast to zero training loss, with their parameters hardly varying. In this work, we show that this "lazy training" phenomenon is not specific to over-parameterized neural networks, and is due to a choice of scaling, often implicit, that makes the model behave as its linearization around the initialization, thus yielding a model equivalent to learning with positive-definite kernels. Through a theoretical analysis, we exhibit various situations where this phenomenon arises in non-convex optimization and we provide bounds on the distance between the lazy and linearized optimization paths. Our numerical experiments bring a critical note, as we observe that the performance of commonly used non-linear deep convolutional neural networks in computer vision degrades when trained in the lazy regime. This makes it unlikely that "lazy training" is behind the many successes of neural networks in difficult high dimensional tasks.

Subfields: Artificial Intelligence, Computational Mechanics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Optimization & Training

Citations in the corpus

Wide neural networks of any depth evolve as linear models under gradient descent (2019, background): “[28]5 argued that infinite width networks are in ‘lazy training’ regime and maybe too simple to be applicable to realistic neural networks.”
High-dimensional dynamics of generalization error in neural networks (2017, background): “However in the active regime, the dynamics remain noninear and can differ dramatically from this linearized behavior, ften attaining better performance (Chizat et al., 2018; Mei, Monanari, & Nguyen, 2018), and presenting a continuing challenge or theoretically understanding generalization properties”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, methodology): “In the lazy training regime, this is well approximated by its first-order Taylor expansion around the initialization w0 (Chizat et al., 2019).”
The large learning rate phase of deep learning: the catapult mechanism (2020, background): “…parameters (Lee et al., 2019); this becomes exact in the inﬁnite width limit, where these dynamics are sometimes called lazy training (Jacot et al., 2018; Lee et al., 2019; Du et al., 2019; Li & Liang, 2018; Zou et al., 2018; Allen-Zhu et al., 2019; Chizat et al., 2019; Dyer & Gur-Ari, 2020).”
The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks (2020, background): “An alternative way to achieve the same effect is to subtract the function output at initialization [Chizat et al., 2019].”
When do neural networks outperform kernel methods? (2020, neutral): “This was termed the ‘lazy regime’ in [9].”
Orthogonal representations for robust context-dependent task performance in brains and neural networks (2022, positive): “As expected from theoretical results (Chizat et al., 2018; Woodworth et al., 2020), the norm of the weights at convergence (Figure 2B, upper) and overall change in inputto-hidden layer weights over learning (Figure 2B, lower) depended strongly on initial connection strengths (Kruskal-Wallis test on ”
Wide Neural Networks Forget Less Catastrophically (2021, background): “Moreover, Chizat et al. (2019) show that these neural networks operate in lazy regime where w ∗ − w in 2 is very small.”
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, background): “The key difference to our work is that their study focuses on the neural tangent kernel (NTK) (Jacot et al., 2018) or “lazy” regime (Chizat et al., 2019) of two-layer networks, where the ﬁrst layer of weights stays approximately constant throughout training.”
Neural Tangents: Fast and Easy Infinite Neural Networks in Python (2019, methodology): “To support these situations, we provide utility functions to efﬁciently compute Monte Carlo estimates of the NNGP covariance and NTK.”
Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks (2020, background): “…a small region around the initial values given sufﬁciently many hidden units - the aptly named “lazy training” regime (Jacot et al., 2018; Li & Liang, 2018; Du et al., 2019a; Arora et al., 2019; Du et al., 2019b; Allen-Zhu et al., 2018; Allen-Zhu & Li, 2019; Zou et al., 2018; Chizat et al., 2019).”
Dash: Semi-Supervised Learning with Dynamic Thresholding (2021, background): “Please note that this is a common property observed in training deep neural networks [Zhang et al., 2017, Allen-Zhu et al., 2019, Du et al., 2019, Arora et al., 2019, Chizat et al., 2019, Hastie et al., 2019, Yun et al., 2019].”
Disentangling trainability and generalization in deep learning (2019, background): “As discussed above, recent work (Jacot et al., 2018; Lee et al., 2019; Chizat et al., 2019) has connected the prior described by the NNGP with the result of gradient descent training using a quantity called the NTK.”
Geometric compression of invariant manifolds in neural nets (2020, background): “For CNNs the feature learning regime tends to perform better [22, 19, 23] but it is not so for fully connected nets using vanilla gradient descent on various benchmarks of images [19].”