Global optimality conditions for deep neural networks

By Chulhee Yun, Suvrit Sra, Ali Jadbabaie. 2017

128 citations recorded by the source corpus.

We study the error landscape of deep linear and nonlinear neural networks with the squared error loss. Minimizing the loss of a deep linear neural network is a nonconvex problem, and despite recent progress, our understanding of this loss surface is still incomplete. For deep linear networks, we present necessary and sufficient conditions for a critical point of the risk function to be a global minimum. Surprisingly, our conditions provide an efficiently checkable test for global optimality, while such tests are typically intractable in nonconvex optimization. We further extend these results to deep nonlinear neural networks and prove similar sufficient conditions for global optimality, albeit in a more limited function space setting.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Citations in the corpus

Visualizing the Loss Landscape of Neural Nets (2017, background): “Theoretical results usually make restrictive assumptions about the sample distributions, non-linearity of the architecture, or loss functions [16, 31, 40, 36, 9, 39].”
Understanding Batch Normalization (2018, neutral): “Under our assumption E (cid:2) (cid:107)∇ (cid:96) i ( x ) − ∇ (cid:96) ( x ) 2 ≤ C we now have”
Generalization in Deep Learning (2017, background): “Another series of theoretical studies have considered how trainable (or optimizable) deep model classes are, revealing their nontrivial properties beyond non-convexity (Choromanska et al., 2015; Kawaguchi, 2016; Yun et al., 2017).”
Optimization Landscape and Expressivity of Deep CNNs (2017, background): “For deep linear networks one has achieved a quite complete picture of the loss surface as it has been shown that every local minimum is a global minimum (Baldi & Hornik, 1988; Kawaguchi, 2016; Freeman & Bruna, 2017; Hardt & Ma, 2017; Yun et al., 2018).”
Distance geometry and data science (2019, background): “Currently, finding “good” optima of ANN training problems is mostly based on experience, although a considerable effort is under way in order to reach a sound definition of optimum [57,198,80,46].”
ResNet with one-neuron hidden layers is a Universal Approximator (2018, background): “Understanding the efficiency of training may require a better understanding of the optimization landscape, a topic of recent attention [5, 16, 24, 26, 8, 35, 27].”
Learning One-hidden-layer ReLU Networks via Gradient Descent (2018, background): “Soudry and Carmon (2016); Xie et al. (2017); Nguyen and Hein (2017); Yun et al. (2017) also showed that there is no spurious local minimum for some other neural networks under stringent assumptions.”
The loss surface and expressivity of deep convolutional neural networks (2017, background): “For deep linear networks one has achieved a quite complete picture of the loss surface as it has been shown that every local minimum is a global minimum (Baldi & Hornik, 1988; Kawaguchi, 2016; Freeman & Bruna, 2017; Hardt & Ma, 2017; Yun et al., 2017).”
A Generalization Theory of Gradient Descent for Learning Over-parameterized Deep ReLU Networks (2019, background): “…2016; Hardt and Ma, 2016; Safran and Shamir, 2017; Xie et al., 2017; Nguyen and Hein, 2017; Soltanolkotabi et al., 2017; Zhou and Liang, 2017; Yun et al., 2017; Du and Lee, 2018; Venturi et al., 2018) investigated the optimization landscape of neural networks with different activation…”
Width Provably Matters in Optimization for Deep Linear Neural Networks (2019, background): “A series of recent papers analyzed landscape of the deep linear network optimization problem (Kawaguchi, 2016; Hardt & Ma, 2016; Lu & Kawaguchi, 2017; Yun et al., 2017; Zhou & Liang, 2018; Laurent & Brecht, 2018).”