The Loss Surface of Deep and Wide Neural Networks

By Quynh N. Nguyen, Matthias Hein. 2017

306 citations recorded by the source corpus.

While the optimization problem behind deep neural networks is highly non-convex, it is frequently observed in practice that training deep networks seems possible without getting stuck in suboptimal points. It has been argued that this is the case as all local minima are close to being globally optimal. We show that this is (almost) true, in fact almost all local minima are globally optimal, for a fully connected network with squared loss and analytic activation function given that the number of hidden units of one layer of the network is larger than the number of training points and the network structure from this layer on is pyramidal.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Visualizing the Loss Landscape of Neural Nets (2017, background): “Theoretical results usually make restrictive assumptions about the sample distributions, non-linearity of the architecture, or loss functions [16, 31, 40, 36, 9, 39].”
Essentially No Barriers in Neural Network Energy Landscape (2018, background): “…and non-convex loss functions are few or based on strong assumptions such as the eigenvalues of the Hessian at critical points being random (Dauphin et al., 2014), linear activations (Choro-manska et al., 2014; Kawaguchi, 2016) or wide hidden layers (Soudry & Carmon, 2016; Nguyen & Hein, 2017).”
Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks (2019, neutral): “…to characterize geometric landscapes of objective functions (Safran & Shamir, 2017; Zhou & Liang, 2017; Freeman & Bruna, 2016; Hardt & Ma, 2016; Nguyen & Hein, 2017; Kawaguchi, 2016; Venturi et al., 2018; Soudry & Carmon, 2016; Du & Lee, 2018; Soltanolkotabi et al., 2018; Haeﬀele & Vidal,…”
Gradient Descent Finds Global Minima of Deep Neural Networks (2018, background): “…activation functions [Soudry and Hoffer, 2017, Safran and Shamir, 2018, 2016, Zhou and Liang, 2017, Freeman and Bruna, 2016, Hardt and Ma, 2016, Nguyen and Hein, 2017, Kawaguchi, 2016, Venturi et al., 2018, Soudry and Carmon, 2016, Du and Lee, 2018, Soltanolkotabi et al., 2018, Haeffele and…”
Efficient Decentralized Deep Learning by Dynamic Model Averaging (2018, neutral): “If the loss function is locally convex in an O (∆)-radius around the current average—a non-trivial but realistic assumption [29, 20]—Theorem 2 in Boley et al.”
Exploring Model-based Planning with Policy Networks (2019, background): “On the other hand, we are inspired by the recent research into understanding deep neural networks [30, 24, 37].”
Gradient Descent with Identity Initialization Efficiently Learns Positive-Definite Linear Transformations by Deep Residual Networks (2018, background): “…treatments of neural network learning algorithms include Lee et al. (1996); Arora et al. (2014); Livni et al. (2014); Janzamin et al. (2015); Safran & Shamir (2016); Zhang et al. (2016); Nguyen & Hein (2017); Zhang et al. (2017); Orhan & Pitkow (2018), although these are less closely related.”
Implicit Bias of Gradient Descent on Linear Convolutional Networks (2018, background): “…sufﬁcient over-parameterization (as is the case with our settings), loss landscape of linear models are well behaved and all local minima are global minima making the problem tractable Burer and Monteiro [2003], Journée et al. [2010], Kawaguchi [2016], Nguyen and Hein [2017], Lee et al. [2016].”
SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data (2017, background): “In a recent work, Nguyen & Hein (2017) show that if training points are linearly separable then under assumptions on the rank of the weight matrices of a fully-connected neural network, every critical point of the loss function is a global minimum.”
Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced (2018, background): “…and Ma, 2016, Bartlett et al., 2018, Haeffele and Vidal, 2015, Freeman and Bruna, 2016, Vidal et al., 2017, Safran and Shamir, 2016, Zhou and Feng, 2017, Nguyen and Hein, 2017a,b, Zhou and Feng, 2017, Safran and Shamir, 2017], showing that the objective functions have properties (ii) and (iii).”
Exponentially vanishing sub-optimal local minima in multilayer neural networks (2017, background): “; This improves over previously known results (Yu, 1992 Huang et al., 2006; Livni et al., 2014; Shen, 2016; Nguyen & Hein, 2017) – which require an extremely wide hidden layer with d 1 ≥ N neurons (and thus Nd 0 parameters) to remove sub-optimal local minima with high probability.”
Training Over-parameterized Deep ResNet Is almost as Easy as Training a Two-layer Network (2019, background): “Many recent works [Choromanska et al., 2015, Kawaguchi, 2016, Nguyen and Hein, 2017, Soudry and Hoﬀer, 2017] have studied the loss surface of the neural networks and a common claim is that (deep) neural networks have relatively no bad local minima.”
Theoretical properties of the global optimizer of two layer neural network (2017, background): “Another line of study looks at the eﬀect of over-parametrization in the training of neural networks [10, 21].”
Global optimality conditions for deep neural networks (2017, result): “Another line of recent results [ 13 – 15 ] provides conditions under which a critical point of the empirical risk is a global minimum.”