Dynamics of Learning Near Singularities in Layered Networks

By Haikun Wei, Jun Zhang, Florent Cousseau, Tomoko Ozeki, Shun-ichi Amari. 2008

84 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Optimization & Training, Statistical Learning Theory, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Frank Nielsen
2. Anima Anandkumar
3. Rong Ge
4. Haikun Wei
5. Ke Sun
6. Shun-ichi Amari
7. Emin Orhan
8. Xaq Pitkow

In Cognitive Neuroscience

1. Shun-ichi Amari
2. Rong Ge
3. Frank Nielsen
4. Anima Anandkumar
5. Xaq Pitkow

In Optimization & Training

1. Haikun Wei
2. Xaq Pitkow
3. Emin Orhan
4. Shun-ichi Amari
5. Rong Ge
6. Frank Nielsen
7. Anima Anandkumar
8. James Martens

In Statistical Learning Theory

1. Frank Nielsen
2. Ke Sun
3. Shun-ichi Amari
4. Rong Ge
5. Anima Anandkumar
6. Xaq Pitkow
7. Yuanzhi Li
8. Zeyuan Allen-Zhu

In Signal Processing

1. Anima Anandkumar
2. Rong Ge
3. Haikun Wei
4. Ke Sun
5. Shun-ichi Amari
6. Frank Nielsen
7. Emin Orhan
8. Xaq Pitkow

Citations in the corpus

• Lightlike Neuromanifolds, Occam's Razor and Deep Learning (2019, background): “Singular regions of the neuromanifold [48] correspond to non-identifiable parameters with rank-deficient FIM, and the learning trajectory typically exhibit chaotic patterns [17] with the singularities which translate into slowdown plateau phenomena when plotting the loss function value against time.”
• Skip Connections Eliminate Singularities (2017, methodology): “The toy model shown in Figure 2e is the simulation of learning dynamics in a network with 3 input, 3 hidden and 3 output units, parametrized in terms of the norms and unit-vector directions of 2008)).”
• Dynamics of learning near singularities in radial basis function networks (2008, methodology): “The singular structure is common to many hierarchical systems such as multilayer perceptrons, RBF networks, Gaussian mixtures, ARMA models of time series, and many others (Amari et al., 2006; Wei et al., 2008).”
• Efficient approaches for escaping higher order saddle points in non-convex optimization (2016, background): “Empirically, the presence of singular saddle points is found to slow down learning substantially [Saad and Solla, 1995, Inoue et al., 2003, Amari et al., 2006, Wei et al., 2008].”
• Data-dependence of plateau phenomenon in learning with neural network—statistical mechanical analysis (2020, unknown): “Plateau phenomenon and singularity in loss landscape as its main cause have been studied by Fukumizu and Amari [2000], Wei et al. [2008], Cousseau et al. [2008] and Guo et al. [2018].”