Kernel and Rich Regimes in Overparametrized Models

By Blake E. Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro H. P. Savarese, Itay Golan, Daniel Soudry, Nati Srebro. 2019

437 citations recorded by the source corpus.

A recent line of work studies overparametrized neural networks in the "kernel regime," i.e. when the network behaves during training as a kernelized linear predictor, and thus training with gradient descent has the effect of finding the minimum RKHS norm solution. This stands in contrast to other studies which demonstrate how gradient descent on overparametrized multilayer networks can induce rich implicit biases that are not RKHS norms. Building on an observation by Chizat and Bach, we show how the scale of the initialization controls the transition between the "kernel" (aka lazy) and "rich" (aka active) regimes and affects generalization properties in multilayer homogeneous models. We also highlight an interesting role for the width of a model in the case that the predictor is not identically zero at initialization. We provide a complete and detailed analysis for a family of simple depth-$D$ models that already exhibit an interesting and meaningful transition between the kernel and rich regimes, and we also demonstrate this transition empirically for more complex matrix factorization models and multilayer non-linear networks.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Statistical Learning Theory

Citations in the corpus

Feature Learning in Infinite-Width Neural Networks (2020, background): “In contrast, (Woodworth et al., 2020) studied a toy class of neural networks in the context of implicit regularization due to the scale α of initialization (which is closely related to logit multiplier of (Chizat & Bach) noted above).”
Orthogonal representations for robust context-dependent task performance in brains and neural networks (2022, positive): “As expected from theoretical results (Chizat et al., 2018; Woodworth et al., 2020), the norm of the weights at convergence (Figure 2B, upper) and overall change in inputto-hidden layer weights over learning (Figure 2B, lower) depended strongly on initial connection strengths (Kruskal-Wallis test on ”
Implicit Regularization via Neural Feature Alignment (2020, background): “By contrast, our aim in this paper is to highlight and discuss non-linear eﬀects, in the (standard) regime where the tangent features and their kernel evolve during training (e.g., Geiger et al., 2019; Woodworth et al., 2020).”
The Pitfalls of Simplicity Bias in Neural Networks (2020, methodology): “Second, the initialization we use is widely studied in the deep learning theory [46, 79] as it better reflects the practical performance of neural networks [14].”
Continual Learning in Linear Classification on Separable Data (2023, background): “Even in the broader deep learning scope, theoreticians often start from very simple models and use them to gain insight into phenomena arising in more practical models (e.g., Belkin et al. (2018); Woodworth et al. (2020)).”
Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks (2019, background): “i=1 `(yi, f(xi)) and LP (f) := E(x,y)∼P [`(y, f(x))] As a concrete example, Woodworth et al. (2019) showed that for matrix completion the NTK solution estimates zero on all unobserved entries and the max-margin solution corresponds to the minimum nuclear norm solution.”
Noise against noise: stochastic label noise helps combat inherent label noise (2021, methodology): “On a quadratically-parameterized model (Vaskevicius et al., 2019; Woodworth et al., 2020), HaoChen et al. (2020) prove that in an over-parameterized regression setting, SGD with label perturbations recovers the sparse groundtruth, whereas SGD with Gaussian noise directly added on gradient descent…”
Implicit Regularization in Deep Learning May Not Be Explainable by Norms (2020, background): “[70] Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro.”
Dynamically Stable Infinite-Width Limits of Neural Classifiers (2020, background): “Nevertheless, a neural tangent kernel of a typical finite-width network evolves significantly: [12] have shown that freezing NTK of practical convolutional nets sufficiently reduces their generalization ability; [13] also noticed that evolution of NTK is sufficient for good performance.”