A Note on Lazy Training in Supervised Differentiable Programming

By Lénaïc Chizat, Francis R. Bach. 2018

158 citations recorded by the source corpus.

Subfields: Optimization & Training, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Statistical Learning Theory

Citations in the corpus

Wide neural networks of any depth evolve as linear models under gradient descent (2019, background): “Mei et al. (2018); Chizat & Bach (2018a); Rotskoff & Vanden-Eijnden (2018); Sirignano & Spiliopoulos (2018) analyze the mean ﬁeld SGD dynamics of training neural networks in the large-width limit.”
Scaling description of generalization with number of parameters in deep learning (2019, neutral): “This result explains why the generalization performance converges as N → ∞, a result previously obtained for single hidden layer neural networks using a different approach [32, 33, 34, 35].”
Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian (2019, background): “This is particularly important in light of a few recent literature [17, 23, 56] suggesting a significant gap between generalization capabilities of kernel methods/linearized neural nets when compared with neural nets operating beyond a linear or NTK learning regime (e.”
Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks (2019, methodology): “…ﬁeld analysis to establish that for inﬁnitely wide neural networks, the empirical distribution of the neural network parameters can be described as a Wasserstein gradient ﬂow (Mei et al., 2018; Chizat & Bach, 2018a; Sirignano & Spiliopoulos, 2018; Rotskoﬀ & Vanden-Eijnden, 2018; Wei et al., 2018).”
On Exact Computation with an Infinitely Wide Neural Net (2019, background): “From a Gaussian process (GP) viewpoint, the correspondence between infinite neural networks and kernel machines was first noted by Neal [1996]. Follow-up work extended this correspondence to more general shallow neural networks [Williams, 1997, Roux and Bengio, 2007, Hazan and Jaakkola, 2015].”
Kernel and Rich Regimes in Overparametrized Models (2019, methodology): “To avoid handling this bias term, and in particular its large scale as α increases, Chizat and Bach [5] suggest using “unbiased” initializations such that F (w0) = 0, so that the bias term vanishes.”
Mean-field Behaviour of Neural Tangent Kernel for Deep Neural Networks (2019, background): “Indeed, recent works have shown that the NTK for ﬁnite width neural networks changes with time (Chizat and Bach, 2018; Ghorbani et al., 2019; Huang and Yau, 2020), and might even be random as shown by (Hanin and Nica, 2019) where authors prove that in the limit n, L → ∞ (where n is a width of the…”
An Improved Analysis of Training Over-parameterized Deep Neural Networks (2019, background): “…number of hidden nodes, to be at least Ω( n 6 /λ 40 ), where n is the training sample size and λ 0 is the smallest eigenvalue of the so-called Gram matrix deﬁned in Du et al. (2018b), which is essentially the neural tangent kernel (Jacot et al., 2018; Chizat and Bach, 2018) on the training data.”
Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel (2018, background): “One line of recent works [17, 37, 22, 21, 2, 75, 31, 6, 16, 71] oﬀers an explanation that invites analogy with kernel methods, proving that with suﬃcient over-parameterization and a certain initialization scale and learning rate schedule, gradient descent essentially learns a linear classiﬁer on…”
Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit (2019, background): “Classical theory of universal approximation provides useful insights into the way two-layers networks capture arbitrary input-output relations Cybenko (1989); Barron (1993).”
Mean Field Analysis of Deep Neural Networks (2019, background): “There is also the related papers [9, 31], whose authors look again at the limit as the units per layer go to infinity, but under a 1 √ N normalization instead of our 1 N normalization.”
Kernel and Deep Regimes in Overparametrized Models (2019, background): “…convergence to zero error solutions in overparametrized settings [7, 8, 1], but it also implies gradient descent will converge to the minimum norm solution (in the corresponding RKHS) [4, 2, 19] and more generally that models will inherit the inductive bias and generalization behaviour of the RKHS.”
Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK (2020, background): “Previous work such as Mei et al. (2018); Chizat and Bach (2018a) show that as the hidden layer width goes to infinity, gradient descent approaches the Wasserstein gradient flow.”
Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems (2019, background): “This setting is the same as those in Du et al. (2019); Arora et al.”
Refined Generalization Analysis of Gradient Descent for Over-parameterized Two-layer Neural Networks with Smooth Activations on Classification Problems (2019, methodology): “3 Brief Review of Functional Gradient Methods Functional gradient methods have been mainly studied for gradient boosting [22, 23] and kernel methods [24, 25, 26, 27, 28] in the machine learning community, but more recently it has been found to be useful in explaining the behavior of gradient descent”
Toward Moderate Overparameterization: Global Convergence Guarantees for Training Shallow Neural Networks (2019, background): “We would like to note that the relationship between kernel methods and deep learning has been emphasized by a few interesting publications [34–37].”
A mean-field limit for certain deep neural networks (2019, background): “[4] Lenaic Chizat and Francis Bach.”