The Loss Surfaces of Multilayer Networks

By Anna Choromanska, Mikael Henaff, Michaël Mathieu, Gérard Ben Arous, Yann LeCun. 2014

1,316 citations recorded by the source corpus.

We study the connection between the highly non-convex loss function of a simple model of the fully-connected feed-forward neural network and the Hamiltonian of the spherical spin-glass model under the assumptions of: i) variable independence, ii) redundancy in network parametrization, and iii) uniformity. These assumptions enable us to explain the complexity of the fully decoupled neural network through the prism of the results from random matrix theory. We show that for large-size decoupled networks the lowest critical values of the random loss function form a layered structure and they are located in a well-defined band lower-bounded by the global minimum. The number of local minima outside that band diminishes exponentially with the size of the network. We empirically verify that the mathematical model exhibits similar behavior as the computer simulations, despite the presence of high dependencies in real networks. We conjecture that both simulated annealing and SGD converge to the band of low critical points, and that all critical points found there are local minima of high quality measured by the test error. This emphasizes a major difference between largeand small-size networks where for the latter poor quality local minima have nonzero probability of being recovered. Finally, we prove that recovering the global minimum becomes harder as the network size increases and that it is in practice irrelevant as global minimum often leads to overfitting.

Subfields: Artificial Intelligence, Statistics and Probability, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Optimization & Training

Citations in the corpus

SGDR: Stochastic Gradient Descent with Warm Restarts (2016, background): “Despite some recent progress in understanding and addressing the latter problems (Bordes et al., 2009; Dauphin et al., 2014; Choromanska et al., 2014; Dauphin et al., 2015), state-of-the-art optimization techniques attempt to approximate the inverse Hessian in a reduced way, e.g., by considering…”
A Closer Look at the Optimization Landscapes of Generative Adversarial Networks (2019, background): “This success has motivated many studies of their non-convex loss landscape (Choromanska et al., 2015; Kawaguchi, 2016; Li et al., 2018b), which, in turn, has led to many improvements, such as better initialization and optimization methods (Glorot and Bengio, 2010; Kingma and Ba, 2015).”
Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields (2017, background): “Even though these are strong assumptions, recent research indicates that the effect of local minima in deep learning vanishes with increasing depth [9, 7, 25], while capacity increases with network size.”
The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “Along this line, an analogy between deep networks and mean-field glasses (p-spins) has been proposed [19], in which the learning dynamics is expected to get stuck in the highest minima of the loss, which are the most abundant.”
Understanding deep learning requires rethinking generalization (2016, background): “But in the linear case, the curvature of all optimal solutions is the same (Choromanska et al., 2015).”
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (2016, background): “Finally, in Section 5 we briefly discuss the relationship between our results and theoretical work [9, 38, 40, 29].”
Train longer, generalize better: closing the generalization gap in large batch training of neural networks (2017, background): “ground breaking results. It has been argued that saddle-points can be avoided (Ge et al., 2015) and that "bad" local minima in the training error vanish exponentially (Dauphin et al., 2014; Choromanska et al., 2015; Soudry & Hoffer, 2017). However, it is still unclear why these complex”
Sharp Minima Can Generalize For Deep Nets (2017, background): “Other works (e.g Dauphin et al., 2014; Choromanska et al., 2015) have looked at the structure of the error surface to analyze how trainable these models are.”
On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization (2018, background): “…linear networks (see for example Kawaguchi (2016); Hardt & Ma (2016) or Baldi & Hornik (1989) for a classic account), or for speciﬁc non-linear networks under different restrictive assumptions ( cf. Choromanska et al. (2015); Haeffele & Vidal (2015); Soudry & Carmon (2016); Safran & Shamir (2017)).”
The Hidden Vulnerability of Distributed Learning in Byzantium (2018, methodology): “…of ﬁnding “the best direction” possible for non–convex cost functions remains one of the most challenging ones in optimization for machine learning (Choromanska et al., 2015b), especially when we stick to such a cheap, ﬁrst– order method as SGD, and avoid expensive, Hessian–like, computations…”
Communication-Efficient Learning of Deep Networks from Decentralized Data (2016, background): “Recent work indicates that in practice, the loss surfaces of sufficiently over-parameterized NNs are surprisingly wellbehaved and in particular less prone to bad local minima than previously thought [11, 17, 9].”
Deep learning with Elastic Averaging SGD (2014, background): “We however emphasize that our problem setting is still highly non-trivial under the communication constraints due to the existence of many local optima [15].”
Visualizing the Loss Landscape of Neural Nets (2017, background): “Numerous theoretical studies have been done on our ability to optimize neural loss function [5, 4].”
Averaging Weights Leads to Wider Optima and Better Generalization (2018, background): “On the other hand Dinh et al. [2017] argue that all the known definitions of sharpness are unsatisfactory and cannot on their own explain generalization.”
Robust Large Margin Deep Neural Networks (2016, background): “The authors in [17] model a loss function of DNN with a spin-glass model and show that for large networks the local optima of the loss function are close to the global optima.”
Recent advances in convolutional neural networks (2015, background): “of parameters and its loss function is non-convex [52], which makes it very difficult to train.”