Local minima and plateaus in hierarchical structures of multilayer perceptrons

By Kenji Fukumizu, Shun-ichi Amari. 2000

245 citations recorded by the source corpus.

Subfields: Analytical Chemistry, Artificial Intelligence, Scaling & Learning Theory, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Analytical Chemistry

In Artificial Intelligence

In Scaling & Learning Theory

In Signal Processing

Citations in the corpus

SGDR: Stochastic Gradient Descent with Warm Restarts (2016, background): “…(Liu & Nocedal, 1989) is not currently in favor in deep learning, not the least due to (i) the stochasticity of ∇ f t ( x t ) , (ii) ill-conditioning of f and (iii) the presence of saddle points as a result of the hierarchical geometric structure of the parameter space (Fukumizu & Amari, 2000).”
No bad local minima: Data independent training error guarantees for multilayer neural networks (2016, background): “data, and are unique in that sense. 2 Related work At ﬁrst, it may seem hopeless to ﬁnd any training error guaran tee for MNNs. Since the loss of MNNs is highly non-convex, with multiple local minima [8], it seems reasonable that optimization with SGD would get stuck at some bad local minimum. Moreo”
Online Batch Selection for Faster Training of Neural Networks (2015, methodology): “Then, we describe our proposed online batch selection procedure and demonstrate that it can substantially speed up both AdaDelta and Adam.”
Exponentially vanishing sub-optimal local minima in multilayer neural networks (2017, background): “Since loss functions parameterized by MNN weights are non-convex, it is unclear why does SGD often work well – rather than converging to sub-optimal local minima with high training error, which are known to exist (Fukumizu & Amari, 2000; Swirszcz et al., 2016).”
Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks (2018, background): “SGD has been analyzed in some cases—for example, when the problem is known to be convex, as in the over-parameterized limit or other idealized settings [8, 9, 10, 11], there are rigorous guarantees of convergence and estimates of convergence rates [12].”
Ellipsoidal Trust Region Methods and the Marginal Value of Hessian Information for Neural Network Training (2019, background): “[32] Kenji Fukumizu and Shun-ichi Amari.”
Dynamics of Learning Near Singularities in Layered Networks (2008, methodology): “Let us consider the following coordinates in the neighborhood of R 1 , which was introduced and used by Fukumizu and Amari (2000) and s is the same as before.”
Quantum Generative Adversarial Networks for learning and loading random distributions (2019, methodology): “[46] J.”
The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks (2021, background): “Fukumizu & Amari (2000) prove that a point corresponding to the global minimum of a smaller model can be a local minimum or a saddle point of the larger model.”
Collapse of Deep and Narrow Neural Nets (2018, methodology): “The authors of (Fukumizu and Amari, 2000) proved that the existence of local minima poses a serious problem in training neural networks.”
Exploring Strategies for Training Deep Neural Networks (2009, background): “The number and quality of these local minima and plateaus (Fukumizu and Amari, 2000) clearly also influence the chances for random initialization to be in the basin of attraction (via gradient descent) of a poor solution.”
Metaheuristic design of feedforward neural networks: A review of two decades of research (2017, background): “It indicates that the critical point corresponding to global minima of a smaller FNN model (model with h − 1 hidden units) can be a local or saddle point of a larger FNN model (model with h hidden units) [47].”
Data-dependence of plateau phenomenon in learning with neural network—statistical mechanical analysis (2020, unknown): “[2000], Fukumizu and Amari [2000]) .”