Gradient descent optimizes over-parameterized deep ReLU networks

290 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Wide Neural Networks Forget Less Catastrophically (2021, background): “Based on the NTK framework, other works (Allen-Zhu et al., 2019; Du et al., 2019; Zou et al., 2020) have proved the convergence of the SGD on various other settings and proved the NTK to be a powerful framework to study neural networks training dynamics.”
Robust early-learning: Hindering the memorization of noisy labels (2021, methodology): “Stochastic gradient descent (SGD) is the most popular one nowadays among the optimization methods (Allen-Zhu et al., 2019; Cao & Gu, 2019; Zou et al., 2020).”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, background): “, 2019), a new line of research is to study the convergence properties of gradient descent (e.g. Allen-Zhu et al., 2019b; Mei and Montanari, 2019; Chizat and Bach, 2018; Du et al., 2018b; Allen-Zhu et al., 2019a; Huang and Yau, 2019; Goldt et al., 2019; Zou et al., 2020; Arora et al., 2019b; Vempala”