Depth Creates No Bad Local Minima

By Haihao Lu, Kenji Kawaguchi. 2017

128 citations recorded by the source corpus.

In deep learning, \textit{depth}, as well as \textit{nonlinearity}, create non-convex loss surfaces. Then, does depth alone create bad local minima? In this paper, we prove that without nonlinearity, depth alone does not create bad local minima, although it induces non-convex loss surface. Using this insight, we greatly simplify a recently proposed proof to show that all of the local minima of feedforward deep linear neural networks are global minima. Our theoretical results generalize previous results with fewer assumptions, and this analysis provides a method to show similar results beyond square loss in deep linear models.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Citations in the corpus

Visualizing the Loss Landscape of Neural Nets (2017, background): “These assumptions have been relaxed by Kawaguchi (2016) and Lu & Kawaguchi (2017), although some assumptions (e.g., of the loss functions) are still required.”
Understanding Batch Normalization (2018, neutral): “Under our assumption E (cid:2) (cid:107)∇ (cid:96) i ( x ) − ∇ (cid:96) ( x ) 2 ≤ C we now have”
Gradient descent aligns the layers of deep linear networks (2018, background): “On the landscape properties of deep linear networks, Lu and Kawaguchi (2017); Laurent and von Brecht (2017) show that under various structural assumptions, all local optima are global.”
Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex (2018, methodology): “On one hand, although recent works have shown the absence of spurious local minimum for deep linear neural networks [50, 35, 44], many of them typically lack theoretical analysis of regularization term.”
Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networks (2018, background): “…are predictors mathematically deﬁned as x (cid:55)→ =1 W i x , with W 1 , . . . , W k being a set of parameter matrices, and k being the depth parameter (e.g. Saxe et al. [2013], Kawaguchi [2016], Hardt and Ma [2016], Lu and Kawaguchi [2017], Bartlett et al. [2018], Laurent and Brecht [2018]).”
Global optimality conditions for deep neural networks (2017, background): “Kawaguchi [ 8 ] and Lu and Kawaguchi [ 11 ] showed that any local minimum is a global minimum, and any other critical points are saddle points.”
Width Provably Matters in Optimization for Deep Linear Neural Networks (2019, background): “A series of recent papers analyzed landscape of the deep linear network optimization problem (Kawaguchi, 2016; Hardt & Ma, 2016; Lu & Kawaguchi, 2017; Yun et al., 2017; Zhou & Liang, 2018; Laurent & Brecht, 2018).”
Deep Linear Networks with Arbitrary Loss: All Local Minima Are Global (2017, background): “Compared to our result, (Lu & Kawaguchi, 2017) therefore allows for the hidden layers of the network to be thinner than the input and output layers.”