Loss landscapes and optimization in over-parameterized non-linear systems and neural networks

By Chaoyue Liu, Libin Zhu, Mikhail Belkin. Applied and Computational Harmonic Analysis, 2020

355 citations recorded by the source corpus.

The success of deep learning is due, to a large extent, to the remarkable effectiveness of gradient-based optimization methods applied to large neural networks. The purpose of this work is to propose a modern view and a general mathematical framework for loss landscapes and efficient optimization in over-parameterized machine learning models and systems of non-linear equations, a setting that includes over-parameterized deep neural networks. Our starting observation is that optimization problems corresponding to such systems are generally not convex, even locally. We argue that instead they satisfy PL$^*$, a variant of the Polyak-Lojasiewicz condition on most (but not all) of the parameter space, which guarantees both the existence of solutions and efficient optimization by (stochastic) gradient descent (SGD/GD). The PL$^*$ condition of these systems is closely related to the condition number of the tangent kernel associated to a non-linear system showing how a PL$^*$-based non-linear theory parallels classical analyses of over-parameterized linear equations. We show that wide neural networks satisfy the PL$^*$ condition, which explains the (S)GD convergence to a global minimum. Finally we propose a relaxation of the PL$^*$ condition applicable to"almost"over-parameterized systems.

Subfields: Artificial Intelligence, Computational Mechanics, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/8da3ed272c07733ca46eab023b03c7411dfcfc42

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hanie Sedghi
2. Behnam Neyshabur
3. Dongsheng Li
4. Tao Sun
5. Bao Wang
6. Nati Srebro
7. Tengyu Ma
8. J. Kolter

In Computational Mechanics

1. Behnam Neyshabur
2. Dongsheng Li
3. Tao Sun
4. Hanie Sedghi
5. Bao Wang
6. Nati Srebro
7. Jason D. Lee
8. Tengyu Ma

In Statistical and Nonlinear Physics

1. Hanie Sedghi
2. Behnam Neyshabur

Citations in the corpus

• Decentralized Federated Averaging (2021, background): “Several examples satisfy Assumption 4 include the loss of training one-hidden-layer networks [45], the loss of training ResNets with linear activation [46], and objective function in matrix factorization [42].”
• The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks (2021, background): “Over-parametrized networks have enough capacity to present different functions that behave similarly on the training data but vastly different on other inputs (Neyshabur et al., 2017; Nguyen et al., 2018; Li et al., 2018; Liu et al., 2020).”