The Loss Surface of Multilayer Networks

By Anna Choromanska, Mikael Henaff, Michaël Mathieu, Gérard Ben Arous, Yann LeCun. 2014

148 citations recorded by the source corpus.

Subfields: Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

SGDR: Stochastic Gradient Descent with Warm Restarts (2016, background): “…and addressing the latter problems (Bordes et al., 2009; Dauphin et al., 2014; Choromanska et al., 2014; Dauphin et al., 2015), state-of-the-art optimization techniques attempt to approximate the inverse Hessian in a reduced way, e.g., by considering only its diagonal to achieve adaptive learning…”
A Theory of Generative ConvNet (2016, background): “It is also believed that getting close to a local mode instead of a global mode may actually prevent over-fitting [3].”
Qualitatively characterizing neural network optimization problems (2014, positive): “These results are consistent with recent empirical and theoretical work arguing that local minima are not a significant problem for training large neural networks (Saxe et al., 2013; Dauphin et al., 2014; Choromanska et al., 2014).”
Essentially No Barriers in Neural Network Energy Landscape (2018, neutral): “, 2014), linear activations (Choromanska et al., 2014; Kawaguchi, 2016) or wide hidden layers (Soudry & Carmon, 2016; Nguyen & Hein, 2017).”
The Shattered Gradients Problem: If resnets are the answer, then what is the question? (2017, background): “The assumption on co-activations is implied by (and so weaker than) the assumption in Choromanska et al. (2015) that activations are Bernoulli random variables independent of the inputs.”
Knowledge Matters: Importance of Prior Information for Optimization (2013, background): “work suggests that it would generally not be because of local minima (Dauphin et al., 2014; Choromanska et al., 2015).”
ℓ1-regularized Neural Networks are Improperly Learnable in Polynomial Time (2015, methodology): “After several simplifying assumptions, a neural network can be shown to be a Gaussian ﬁeld whose critical points can be analyzed using the Kac-Rice formula and properties of the Gaussian Orthogonal Ensemble [3, 10, 8].”
Online Batch Selection for Faster Training of Neural Networks (2015, background): “Despite some recent progress in understanding and addressing the latter problems (Bordes et al., 2009; Dauphin et al., 2014; Choromanska et al., 2014; Dauphin et al., 2015), state-of-the-art optimization techniques tend to approximate the inverse Hessian in a reduced way, e.g., by considering only…”
RMSProp and equilibrated adaptive learning rates for non-convex optimization (2015, positive): “variables only exists3 when the Hessian H is positive semi-deﬁnite. This is a problem for non-convex loss surfaces where the Hessian might be indeﬁnite. In fact, recent studies (Dauphin et al., 2014; Choromanska et al., 2014) has shown that saddle points are dominating the optimization landscape of ”
On the energy landscape of deep networks (2015, background): “Our work is closely related to Choromanska et al. (2014) which discusses the exponential regime for a spin glass model of dense, deep networks under assumptions where paths in the network connected to different input neurons are independent.”
Deep Online Convex Optimization by Putting Forecaster to Sleep (2015, methodology): “However, recent work suggests that most optima in rectifier neural networks are good enough (Choromanska et al., 2014).”
On the Learning Dynamics of Deep Neural Networks (2018, background): “Choromanska et al. (2014); Raghu et al. (2017); Saxe (2015); Yosinski et al. (2014) also focus on neural network dynamics, while Nacson et al. (2018); Xu et al. (2018); Soudry et al. (2017) study the convergence rate of learning on separable data.”
Theoretical properties of the global optimizer of two layer neural network (2017, background): “E .g., [5] made a few assumptions to show that every local minimum of the simpliﬁed objective is close to the global minimum.”
Deep Q-Networks for Accelerating the Training of Deep Neural Networks (2016, background): “It has been argued in Choromanska et al. (2014) that there are many local minima in the loss surfaces for large-sized deep models and they have similar performance on a test dataset.”
The Goldilocks zone: Towards better understanding of neural network loss landscapes (2018, background): “The general structure of the objective landscape has been a subject of a large number of studies (Choromanska et al. 2014; Keskar et al. 2016).”
Gradient Descent Converges to Minimizers (2016, background): “We remark that the strict saddle assumption is necessary in the worst case, due to hardness results regarding testing the local optimality of functions whose Hessians are highly degenerate at criticalpoints (e.g, quartic polynomials) [19].”
On the Quality of the Initial Basin in Overspecified Neural Networks (2015, background): “Another relevant work is (Choromanska et al., 2014), which also investigates the objective function of ReLU networks.”
Explorations on high dimensional landscapes (2014, background): “In a slightly different approach Dauphin et al. (2014) and Choromanska et al. (2014) exhibit that stochastic gradient descent might actually get stuck in critical points other than local minima.”