The implicit bias of gradient descent on nonseparable data

By Ziwei Ji, Matus Telgarsky. 2019

208 citations recorded by the source corpus.

Subfields: Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks (2020, background): “…has also been studied in various settings, suggesting a bias towards large-margin, low-norm, or low-rank solutions [Gunasekar et al., 2017, 2018, Soudry et al., 2018, Li et al., 2018, Ji and Telgarsky, 2019a,b, Arora et al., 2019a, Lyu and Li, 2019, Chizat and Bach, 2020, Razin and Cohen, 2020].”
The Implicit Bias of Gradient Descent on Separable Data (2017, background): “Degenerate and Non-Degenerate Data Sets An earlier conference version of this paper (Soudry et al., 2018) included a partial version of Theorem 3, which only applies to almost all data sets, in which case we can ensure the residual ρ ( t ) is bounded.”
The Pitfalls of Simplicity Bias in Neural Networks (2020, background): “Recent works [67, 35] also analyze the limiting direction of gradient descent on logistic regression with linearly separable and non-separable data respectively.”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, background): “The same phenomenon is referred to as implicit bias, implicit regularization, simplicity bias and spectral bias in several works [83, 75, 36, 74, 70, 53, 94, 10, 13, 35, 82, 66].”
Classification vs regression in overparameterized regimes: Does the loss function matter? (2020, methodology): “Thus, characterizing the implicit regularization [23, 38, 21, 45, 30] induced by the choice of optimization algorithm is important to understand properties of the obtained solutions.”
Gradient Descent Maximizes the Margin of Homogeneous Neural Networks (2019, background): “For gradient flow, similar results on the convergent direction are known for linear fully-connected networks [Ji and Telgarsky, 2019].”
SGD on Neural Networks Learns Functions of Increasing Complexity (2019, background): “We provide evidence for this conjecture in Section [7, 14, 33, 13, 24, 34, 19].”