Kernel and Deep Regimes in Overparametrized Models

By Blake E. Woodworth, Suriya Gunasekar, Jason D. Lee, Daniel Soudry, Nati Srebro. 2019

33 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

Citations in the corpus

The large learning rate phase of deep learning: the catapult mechanism (2020, background): “…limit of inﬁnite width (Daniely, 2017; Jacot et al., 2018; Lee et al., 2019; Du et al., 2019; Zou et al., 2018; Allen-Zhu et al., 2019; Li & Liang, 2018; Chizat et al., 2019; Mei et al., 2018; Rotskoff & Vanden-Eijnden, 2018; Sirignano & Spiliopoulos, 2018; Woodworth et al., 2019; Naveh et al.).”
Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks (2019, background): “A large body of work Li et al. (2017); Nacson et al. (2019); Gunasekar et al. (2018b;a; 2017); Woodworth et al. (2019) shows that asymptotically gradient descent converges to a max-margin solution with a strong regularization effect, unlike the NTK regularization 1 .”
Generalization of Two-layer Neural Networks: An Asymptotic Viewpoint (2020, background): “Williams et al. (2019); Woodworth et al. (2019); Li et al. (2017) showed that for certain two-layer network or overparameterized linear model, the scale of initialization controls the implicit regularization of gradient descent.”
Dynamically Stable Infinite-Width Limits of Neural Classifiers (2020, background): “Nevertheless, a neural tangent kernel of a typical finite-width network evolves significantly: [11] have shown that freezing NTK of practical convolutional nets sufficiently reduces their generalization ability; [12] also noticed that evolution of NTK is sufficient for good performance.”
Classification vs regression in overparameterized regimes: Does the loss function matter? (2020, background): “Thus, characterizing the implicit regularization [23, 38, 21, 45, 30] induced by the choice of optimization algorithm is important to understand properties of the obtained solutions.”
Frequency Bias in Neural Networks for Input of Non-Uniform Density (2020, background): “Recent theoretical and empirical results suggest that NTK models still somewhat underperform common nonlinear networks (Arora et al., 2019a; Chizat et al., 2019; Novak et al., 2019; Woodworth et al., 2019).”
The Implicit and Explicit Regularization Effects of Dropout (2020, background): “Another line of work studies implicit regularization in deep learning (see (Gunasekar et al., 2017, 2018a,b; Soudry et al., 2018; Woodworth et al., 2019; Arora et al., 2019) and references therein).”