Implicit Bias of Gradient Descent on Linear Convolutional Networks

By Suriya Gunasekar, Jason D. Lee, Daniel Soudry, Nati Srebro. 2018

483 citations recorded by the source corpus.

We show that gradient descent on full-width linear convolutional networks of depth $L$ converges to a linear predictor related to the $\ell_{2/L}$ bridge penalty in the frequency domain. This is in contrast to linearly fully connected networks, where gradient descent converges to the hard margin linear support vector machine solution, regardless of depth.

Subfields: Artificial Intelligence, Computational Mechanics, Computational Theory and Mathematics, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computational Theory and Mathematics

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

Implicit Regularization of Discrete Gradient Dynamics in Deep Linear Neural Networks (2019, background): “…finite number n of inputs xi ∈ Rd , 1 ≤ i ≤ n we want to predict a multi-dimensional outputs yi ∈ Rp , 1 ≤ i ≤ n with a deep linear network [Saxe et al., 2018, Gunasekar et al., 2018], Deep linear model: ŷd(x) := W>L · · ·W>1 x , (1) whereW1, . . . ,WL are learned through a MSE formulation with…”
The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks (2020, background): “…of (stochastic) gradient descent has also been studied in various settings, suggesting a bias towards large-margin, low-norm, or low-rank solutions [Gunasekar et al., 2017, 2018, Soudry et al., 2018, Li et al., 2018, Ji and Telgarsky, 2019a,b, Arora et al., 2019a, Lyu and Li, 2019, Chizat and…”
Initialization and Regularization of Factorized Neural Layers (2021, background): “We are not the ﬁrst study gradient descent on factorized layers; in particular, deep linear nets are well-studied in theory (Saxe et al., 2014; Gunasekar et al., 2019).”
Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian (2019, background): “For linearly separable datasets, [55, 27, 14, 32, 33] show that first-order methods find solutions that generalize well without an explicit regularization for logistic regression.”
Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks (2019, methodology): “…established that ﬁrst-order methods can automatically ﬁnd minimum-norm/maximum-margin solutions that ﬁt the data in the settings of logistic regression, deep linear networks, and symmetric matrix factorization (Soudry et al., 2018; Gunasekar et al., 2018a,b; Ji & Telgarsky, 2018; Li et al., 2018b).”
The Implicit Bias of Gradient Descent on Separable Data (2017, methodology): “In Section 4.3 and in follow-up work (Gunasekar et al., 2018) we discuss also other optimization algorithms, and their implied biases.”
Stochastic Mirror Descent on Overparameterized Nonlinear Models (2019, background): “In this case, the model is f , and the loss is The following result characterizes the solution that SMD converges to (Azizan & Hassibi, 2019; Gunasekar et al., 2018a).”
Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel (2018, background): “Other papers [26, 68, 46, 28, 38, 27, 38, 32] study implicit regularization towards a specific solution.”
Generalization bounds for deep convolutional neural networks (2019, background): “(For some deeper theoretical exploration of implicit bias in deep learning and related settings, see (Gunasekar et al., 2017; 2018a;b; Ma et al., 2018).)”
Size-free generalization bounds for convolutional neural networks (2019, background): “[12] Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro.”
Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks (2019, background): “A large body of work Li et al. (2017); Nacson et al. (2019); Gunasekar et al. (2018b;a; 2017); Woodworth et al. (2019) shows that asymptotically gradient descent converges to a max-margin solution with a strong regularization effect, unlike the NTK regularization 1 .”
A Generalization Theory of Gradient Descent for Learning Over-parameterized Deep ReLU Networks (2019, background): “A bunch of papers (Gunasekar et al., 2017; Soudry et al., 2017; Ji and Telgarsky, 2018; Gunasekar et al., 2018a,b; Nacson et al., 2018; Li et al., 2018b) have studied implicit regularization/bias of GD, stochastic gradient descent (SGD) or mirror descent for matrix factorization, logistic…”
Why Do Better Loss Functions Lead to Less Transferable Features? (2020, background): “These results can be extended to neural networks in certain restricted settings [28, 74].”
Kernel and Deep Regimes in Overparametrized Models (2019, methodology): “…on deep linear convolutional networks implicitly biases toward minimizing the L p bridge penalty, for p = 2 / depth ≤ 1 , in the frequency domain [13]; weight decay on an inﬁnite width single input ReLU implicitly biases towards minimizing the second order total variations | f (cid:48)(cid:48) (…”
Why do Larger Models Generalize Better? A Theoretical Perspective via the XOR Problem (2018, background): “Several works (Ji & Telgarsky, 2019; Gunasekar et al., 2018; Arora et al., 2019) provide guarantees for gradient descent on linear networks.”
On the Margin Theory of Feedforward Neural Networks (2018, background): “Other papers (Gunasekar et al., 2017; Soudry et al., 2018; Nacson et al., 2018; Gunasekar et al., 2018b; Li et al., 2018; Gunasekar et al., 2018a) study implicit regularization towards a speciﬁc solution.”
The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies (2019, unknown): “[12, 15] provide related results.”
Generalization of Two-layer Neural Networks: An Asymptotic Viewpoint (2020, result): “Existing works have analyzed the implicit regularization of gradient descent (Gunasekar et al., 2018; Ji and Telgarsky, 2018), and provided generalization guarantees (Arora et al., 2018; Bartlett et al., 2017; Dziugaite and Roy, 2017) that align with the empirical observations.”
The Platonic Representation Hypothesis (2024, background): “…in the absence of external influences, deep networks naturally adhere to Occam’s razor, implicitly favoring simple solutions that fit the data (Solomonoff, 1964; Gunasekar et al., 2018; Arora et al., 2019a; Valle-Perez et al., 2019; Huh et al., 2023; Din-gle et al., 2018; Goldblum et al., 2023).”
Implicit Regularization in Deep Learning May Not Be Explainable by Norms (2020, background): “[34, 36, 8]) to distinguish between implicit regularization — which concerns the type of solutions found in training — and the complementary question of whether training loss is globally optimized.”