A Generalization Theory of Gradient Descent for Learning Over-parameterized Deep ReLU Networks

By Yuanbin Cao, Quanquan Gu. 2019

71 citations recorded by the source corpus.

Subfields: Optimization & Training, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Statistical Learning Theory

Citations in the corpus

Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian (2019, background): “This is in stark contrast to related works [4, 21, 2, 15] where network size grows inversely proportional to the distance between the input samples or other notions of margin.”
On Lazy Training in Differentiable Programming (2018, background): “Other works in this line of research have appeared since the first version of this article was communicated, studying the optimization and statistical properties of various neural networks architectures in what we refer to as the lazy regime [5, 33, 32, 29, 46, 8, 27, 16, 4, 43].”
On the Convergence Rate of Training Recurrent Neural Networks (2018, methodology): “3)); and our techniques in Section 5 give a general theory for the stability of multi-layer networks against adversarial weight perturbations, which is at the heart of showing the semi-smoothness Theorem 4, and used by all the follow-up works [1–3, 10].”
Stochastic Mirror Descent on Overparameterized Nonlinear Models (2019, background): “…few years to study deep learning from an optimization perspective, e.g., (Achille & Soatto, 2017; Chaudhari & Soatto, 2018; Shwartz-Ziv & Tishby, 2017; Allen-Zhu et al., 2019; Oymak & Soltanolkotabi, 2019; Azizan & Hassibi, 2019; Ma et al., 2018b; Du et al., 2018; Li & Liang, 2018; Cao & Gu, 2019).”
A type of generalization error induced by initialization in deep neural networks (2019, background): “The generalization error bounds (Arora et al. 2019, Cao & Gu 2019, E, Ma, Wang & Wu 2019, E, Ma & Wu 2019) are further studied in the kernel regime.”
Refined Generalization Analysis of Gradient Descent for Over-parameterized Two-layer Neural Networks with Smooth Activations on Classification Problems (2019, background): “As seen in Table 1, for more comprehensive sizes of two-layer networks with respect to the network width, our theory ensures the same generalization ability with respect to n and T as that of [8].”
Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks (2019, methodology): “Following the same standard setup implemented in the line of recent work [2, 11, 39, 8], we consider fully connected neural networks with width m, depth L and input dimension d.”
Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation (2019, background): “Finally, many recent papers have shown optimization results for neural nets (Allen-Zhu et al., 2018a,b; Li & Liang, 2018; Arora et al., 2019; Mei et al., 2018; Zou et al., 2018; Du et al., 2018b,a; Lee et al., 2019; Jacot et al., 2018; Cao & Gu, 2019).”
A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics (2019, background): “Similar results were proved in [20] for the classification problem, in [8] for offline SGD algorithms, and in [1] for GD algorithm.”
On the Convergence and Robustness of Adversarial Training (2021, background): “While ReLU (Nair & Hinton, 2010) is non-differentiable, recent studies (Allen-Zhu et al., 2018; Du et al., 2018; Zou et al., 2018; Cao & Gu, 2019) showed that the loss function of overparamterized deep neural networks is semi-smooth.”
On the Power and Limitations of Random Features for Understanding Neural Networks (2019, methodology): “[23], [15], [3], [9], [2], [1]) also use the method of "coupling" and the popular ReLU activation.”
Learning One-hidden-layer ReLU Networks via Gradient Descent (2018, background): “, 2018), and their generalization performance (Li and Liang, 2018; Allen-Zhu et al., 2018a; Cao and Gu, 2019), in the over-parameterized regime where the width of the neural network is much larger than the training set size.”
Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks (2019, background): “Theoretically, a line of work study the optimization and generalization behavior of ultra-wide NNs (Allen-Zhu et al., 2018b,a; Arora et al., 2019b; Du et al., 2018b,a; Li & Liang, 2018; Zou et al., 2018; Yang, 2019; Cao & Gu, 2019a,b).”
Generalization of Two-layer Neural Networks: An Asymptotic Viewpoint (2020, background): “On the other hand, Jacot et al. (2018); Du et al. (2018); Arora et al. (2019a); Oymak and Soltanolkotabi (2019); Cao and Gu (2019); Allen-Zhu et al. (2018b); Song and Yang (2019) considered a diﬀerent scaling and showed that gradient descent converges to global minimizer at a linear rate; key to…”