Geometry of energy landscapes and the optimizability of deep neural networks
By Simon Becker, Yao Zhang, Alpha Albert Lee. 2018
33 citations recorded by the source corpus.
Deep neural networks are workhorse models in machine learning with multiple layers of nonlinear functions composed in series. Their loss function is highly nonconvex, yet empirically even gradient descent minimization is sufficient to arrive at accurate and predictive models. It is hitherto unknown why deep neural networks are easily optimizable. We analyze the energy landscape of a spin glass model of deep neural networks using random matrix theory and algebraic geometry. We analytically show that the multilayered structure holds the key to optimizability: Fixing the number of parameters and increasing network depth, the number of stationary points in the loss function decreases, minima become more clustered in parameter space, and the trade-off between the depth and width of minima becomes less severe. Our analytical results are numerically verified through comparison with neural networks trained on a set of classical benchmark datasets. Our model uncovers generic design principles of machine learning models.
Subfields: Artificial Intelligence, Computational Mathematics, Statistical and Nonlinear Physics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Medhi Bennis
- Jihong Park
- Sumudu Samarakoon
- Merouane Debbah
- Anis Elgabli
- H. Vincent Poor
- Seong-Lyun Kim
- Walid Saad
In Statistical and Nonlinear Physics
Citations in the corpus
- Wireless Network Intelligence at the Edge (2018, background): “Exploiting this connection, a recent discovery [87] verifies that the number of Hamiltonian’s critical points is a positively skewed unimodal curve over L.”