A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics

By E. Weinan, Chao Ma, Lei Wu. 2019

132 citations recorded by the source corpus.

A fairly comprehensive analysis is presented for the gradient descent dynamics for training two-layer neural network models in the situation when the parameters in both layers are updated. General initialization schemes as well as general regimes for the network width and training data size are considered. In the over-parametrized regime, it is shown that gradient descent dynamics can achieve zero training loss exponentially fast regardless of the quality of the labels. In addition, it is proved that throughout the training process the functions represented by the neural network model are uniformly close to that of a kernel method. For general values of the network width and training data size, sharp estimates of the generalization error is established for target functions in the appropriate reproducing kernel Hilbert space.

Subfields: Artificial Intelligence, Optimization & Training, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Statistical Learning Theory

Citations in the corpus

Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian (2019, background): “This is in stark contrast to recent literature on optimization and generalization of neural networks [4, 21, 2, 15, 40, 1, 14] where guarantees only hold for very wide networks with the width of the network growing inversely proportional to the distance between the input samples or class margins or ”
On Lazy Training in Differentiable Programming (2018, background): “Other works in this line of research have appeared since the first version of this article was communicated, studying the optimization and statistical properties of various neural networks architectures in what we refer to as the lazy regime [5, 33, 32, 29, 46, 8, 27, 16, 4, 43].”
A Theoretical Analysis of Deep Q-Learning (2019, background): “model (Rahimi and Recht, 2008, 2009). See, e.g., Du et al. (2019b,a); Zou et al. (2018); Chizat et al. (2019); Allen-Zhu et al. (2019a,b); Jacot et al. (2018); Cao and Gu (2019); Arora et al. (2019); Weinan et al. (2019); Mei et al. (2019); Yehudai and Shamir (2019); Bietti and Mairal (2019); Yang a”
Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks (2019, methodology): “Our generalization bound can also be compared with two recent results (Yehudai and Shamir, 2019; E et al., 2019) for two-layer neural networks.”