Explorations on high dimensional landscapes

By Levent Sagun, V. U. Güney, Yann LeCun. 2014

64 citations recorded by the source corpus.

Finding minima of a real valued non-convex function over a high dimensional space is a major challenge in science. We provide evidence that some such functions that are defined on high dimensional domains have a narrow band of values whose pre-image contains the bulk of its critical points. This is in contrast with the low dimensional picture in which this band is wide. Our simulations agree with the previous theoretical work on spin glasses that proves the existence of such a band when the dimension of the domain tends to infinity. Furthermore our experiments on teacher-student networks with the MNIST dataset establish a similar phenomenon in deep networks. We finally observe that both the gradient descent and the stochastic gradient descent methods can reach this level within the same number of steps.

Subfields: Global and Planetary Change, Transportation, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Global and Planetary Change

In Transportation

In Signal Processing

Citations in the corpus

AdaNet: Adaptive Structural Learning of Artificial Neural Networks (2016, background): “(e.g. [Choromanska et al., 2014, Sagun et al., 2014, Zhang et al., 2015, Livni et al., 2014]).”
Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond (2016, background): “This work is part of a series of papers that explore the data-model-algorithm connection along with Sagun et al. (2014; 2015) and it builds on top of the intuition developed in LeCun et al. (2012).”
Topology and Geometry of Half-Rectified Network Optimization (2016, background): “When κ = 0, it has been shown in Saxe et al. (2013) and Kawaguchi (2016) that in this case, every local minima is a global minima.”
A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks (2019, background): “Even though SGD seems to find zero loss solutions on the training landscape (at least in certain regimes (Zhang et al., 2017a; Sagun et al., 2015; Keskar et al., 2016; Geiger et al., 2018)), it appears that the algorithm finds solutions with different properties depending on how it is tuned (Sutskev”
On the energy landscape of deep networks (2015, background): “The energy landscape also has a layered structure, i.e., high-order saddle points with many negative eigenvalues of the Hessian lie at high energies while saddle points Computer Science Department, University of California, Los Angeles.”
Sharp Minima Can Generalize For Deep Nets (2017, background): “Other works (e.g Dauphin et al., 2014; Sagun et al., 2014; Choromanska et al., 2015) have looked at the structure of the error surface to analyze how trainable these models are.”
Trainability and Accuracy of Artificial Neural Networks: An Interacting Particle System Approach (2018, background): “The function (5.1) is known to have a number of critical points that grows exponentially with the dimensionality d [2,3,29].”
Comparing dynamics: deep neural networks versus glassy systems (2018, background): “In fact, simulations in Sagun et al. (2014) demonstrate that different dynamical processes on the loss landscape can indeed perform similarly regardless of the effect of the noise of SGD, thus suggesting that barrier crossing indeed does not take place.”
Local minima in training of deep networks (2017, unknown): “Similar claim is put forward in Sagun et al. (2014).”
Local minima in training of neural networks (2016, unknown): “Similar claim is put forward in Sagun et al. (2014).”
Singularity of the Hessian in Deep Learning (2016, background): “This work is part of a series of papers that explore the data-model-algorithm connection along with Sagun et al. (2014; 2015) and it builds on top of the intuition developed in Lecun et al. (1998).”
The Effect of Gradient Noise on the Energy Landscape of Deep Networks (2015, background): “Sagun et al. (2014) show that SGD fails to progress beyond a very specific energy barrier, one that can be seen in spin glass computations as the onset of high-order saddle points and local minima.”
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks (2017, methodology): “We present numerical evidence and mathematical justifications to the following conjectures laid out by Sagun et al. [2016]: Fixing data, increasing the number of parameters merely scales the bulk of the spectrum; fixing the dimension and changing the data (for instance adding more clusters or making”
The committee machine: computational to statistical gaps in learning a two-layers neural network (2018, background): “Similar large computational gaps have been previously identified in many problems and we believe that its identification in a multi-layer neural network model makes it a very interesting candidate for further mathematical studies of the energy landscape in deep learning [12, 47].”
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning (2018, methodology): “This includes: work on large-batch learning and the generalization gap [148, 72, 64, 57, 67, 131, 66, 146, 94, 151, 152]; work on Energy Landscape approaches to NN training [70, 149, 44, 123, 30, 29, 27, 65, 47, 12, 104, 150, 50, 83, 82, 95]; work on using weight matrices or properties of weight mat”
Neural Networks as Interacting Particle Systems: Asymptotic Convexity of the Loss Landscape and Universal Scaling of the Approximation Error (2018, background): “The function (4.1) is known to have a number of critical points that grows exponentially with the dimensionality d [1, 2, 24].”