Statistics of critical points of Gaussian fields on large-dimensional spaces.

By A. Bray, D. Dean. 2006

183 citations recorded by the source corpus.

We calculate the average number of critical points of a Gaussian field on a high-dimensional space as a function of their energy and their index. Our results give a complete picture of the organization of critical points and are of relevance to glassy and disordered systems and landscape scenarios coming from the anthropic approach to string theory.

Subfields: Atmospheric Science, Condensed Matter Physics, Mathematical Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Atmospheric Science

In Condensed Matter Physics

In Mathematical Physics

Citations in the corpus

Entropy-SGD: biasing gradient descent into wide valleys (2016, background): “They also relate to earlier studies such as Baldi & Hornik (1989); Fyodorov & Williams (2007); Bray & Dean (2007) which show that critical points with high training error are exponentially likely to be saddle points with many negative directions and all local minima are likely to have error that is…”
Troubling Trends in Machine Learning Scholarship (2018, background): “As one piece of evidence, the work cites a statistical physics paper [9] on Gaussian random fields and states that in high dimensions “all local minima [of Gaussian random fields] are likely to have an error very close to that of the global minimum” (a similar statement appears in the related work o”
On the saddle point problem for non-convex optimization (2014, background): “As learning progresses, the norm of the negative eigenvalue decreases as it is predicted by the theory of random matrices Bray and Dean (2007) (think of the semi-circular distribution being shifted to the right).”
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization (2014, background): “This empirical test confirms that the observations by Bray and Dean (2007) qualitatively hold for neural networks.”
Train longer, generalize better: closing the generalization gap in large batch training of neural networks (2017, methodology): “…local minimas tend to have low error using an analogy between L ( w ) , the DNN loss surface, and the high-dimensional Gaussian random ﬁeld analyzed in Bray & Dean (2007), which had zero mean and auto-covariance for some function f , where the expectation now is over the randomness of the loss.”
Extreme value statistics of correlated random variables: A pedagogical review (2019, background): “This is of interest in a variety of fields [259], including population dynamics [260, 261], models of evolutionary biology [54, 262], spin-glasses [215, 263–271], neural networks [272] as well as in landscape based string theory [273, 274] and cosmology [275].”
The Loss Surfaces of Multilayer Networks (2014, background): “Earlier theoretical analyses [Baldi and Hornik, 1989, Wigner, 1958, Fyodorov and Williams, 2007, Bray and Dean, 2007] suggest the existence of a certain structure of critical points of random Gaussian error functions on high dimensional continuous spaces.”
An Empirical Analysis of Deep Network Loss Surfaces (2016, methodology): “One approach is to analogize the states of neurons as the magnetics dipoles used in spherical spinglass Ising models from statistical physics (Parisi, 2016; Fyodorov & Williams, 2007; Bray & Dean, 2007).”
Global optimization issues in deep network regression: an overview (2018, background): “This idea has been confirmed in [30] where it is proved that in a critical point ω̂ there is a strong correlation between the value of the training error E(ω̂) versus the number of negative eigenvalues of the hessian∇2E(ω̂); in particular the larger the error, the greater the number of negative eige”