Local minima in training of neural networks

By Grzegorz Swirszcz, Wojciech M. Czarnecki, Razvan Pascanu. 2016

82 citations recorded by the source corpus.

There has been a lot of recent interest in trying to characterize the error surface of deep models. This stems from a long standing question. Given that deep networks are highly nonlinear systems optimized by local gradient methods, why do they not seem to be affected by bad local minima? It is widely believed that training of deep models using gradient methods works so well because the error surface either has no local minima, or if they exist they need to be close in value to the global minimum. It is known that such results hold under very strong assumptions which are not satisfied by real models. In this paper we present examples showing that for such theorem to be true additional assumptions on the data, initialization schemes and/or the model classes have to be made. We look at the particular case of finite size datasets. We demonstrate that in this scenario one can construct counter-examples (datasets or initialization schemes) when the network does become susceptible to bad local minima over the weight space.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Visualizing the Loss Landscape of Neural Nets (2017, background): “Swirszcz et al. (2017) show counterexamples that achieve “bad” local minima for toy problems.”
Spurious Local Minima are Common in Two-Layer ReLU Neural Networks (2017, background): “(1)), there are many spurious local minima (e.g., [Auer et al., 1996, Swirszcz et al., 2016]).”
Exponentially vanishing sub-optimal local minima in multilayer neural networks (2017, background): “Since loss functions parameterized by MNN weights are non-convex, it is unclear why does SGD often work well – rather than converging to sub-optimal local minima with high training error, which are known to exist (Fukumizu & Amari, 2000; Swirszcz et al., 2016).”
Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory (2019, background): “Such results exist for simple fully connected architectures (Swirszcz et al., 2016), single-layer networks (Liang et al., 2018; Yun et al., 2018), and two-layer ReLU networks (Safran & Shamir, 2017).”
Spurious Valleys in One-hidden-layer Neural Network Optimization Landscapes (2019, methodology): “…(Safran and Shamir, 2017) showed counterexamples under Gaussian input distributions, for p = n − 1 ∈ {8, . . . , 19}, using a computer-assisted proof; (Swirszcz et al., 2016) and (Zhou and Liang, 2017) provided a few numerical examples; (Yun et al., 2018) showed existence of spurious minima for…”
Depth Creates No Bad Local Minima (2017, unknown): “Kawaguchi, 2016; Swirszcz et al., 2016; Arora et al., 2016; Freeman & Bruna, 2016; Soudry & Hoffer, 2017).”
Backtracking gradient descent method for general $C^1$ functions, with applications to Deep Learning (2018, background): “…dimensional intuition, a working hypothesis in Deep Learning is that local minima with high error (that is, a minimum point at which the value of the function is too much bigger than the optimal value) are exponentially rare in higher dimensions [11, 45], and hence ﬁnding local minima is enough.”