Training Over-parameterized Deep ResNet Is almost as Easy as Training a Two-layer Network

By Huishuai Zhang, Da Yu, Tie-Yan Liu. 2019

15 citations recorded by the source corpus.

Subfields: Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

1. Quanquan Gu
2. Lénaïc Chizat
3. Francis R. Bach
4. Difan Zou
5. Edouard Oyallon
6. Jason D. Lee
7. Julien Mairal
8. Zeyuan Allen-Zhu

In Scaling & Learning Theory

1. Francis R. Bach
2. Difan Zou
3. Edouard Oyallon
4. Lénaïc Chizat
5. Quanquan Gu
6. Aymeric Dieuleveut
7. Sebastian U. Stich
8. Kumar Kshitij Patel

Citations in the corpus

• Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian (2019, neutral): “These works build on global convergence results of randomly initialized neural networks [21, 20, 2, 17, 60, 46, 47, 61] which study the gradient descent trajectory via comparisons to a a linearized Neural Tangent Kernel (NTK) learning problem.”
• On Lazy Training in Differentiable Programming (2018, background): “Other works in this line of research have appeared since the first version of this article was communicated, studying the optimization and statistical properties of various neural networks architectures in what we refer to as the lazy regime [5, 33, 32, 29, 46, 8, 27, 16, 4, 43].”
• An Improved Analysis of Training Over-parameterized Deep Neural Networks (2019, methodology): “…can be also applied to prove similar results for other loss functions such as cross-entropy loss and other neural network architectures such as convolutional neural networks (CNN) (Allen-Zhu et al., 2018b; Du et al., 2018b) and ResNet (Allen-Zhu et al., 2018b; Du et al., 2018b; Zhang et al., 2019).”