Neural networks and principal component analysis: Learning from examples without local minima

By Pierre Baldi, Kurt Hornik. 1989

Subfields: Representation & Self-Supervised Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Model Architectures

Citations in the corpus

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013, background): “The fixed point structure of gradient descent learning was worked out in [19].”
High-dimensional dynamics of generalization error in neural networks (2017, background): “…a simple input-output map, deep linear networks retain several important features of nonlinear networks: most notably, the learning problem is nonconvex (P. Baldi and Hornik, 1989; Kawaguchi, 2016), yielding nonlinear dynamical trajectories (Fukumizu, 1998; Saxe, McClelland, and Ganguli, 2014).”
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization (2014, background): “Baldi and Hornik (1989) analyzed the error surface of a multilayer perceptron (MLP) with a single linear hidden layer.”
The Loss Surfaces of Multilayer Networks (2014, background): “Earlier theoretical analyses [Baldi and Hornik, 1989, Wigner, 1958, Fyodorov and Williams, 2007, Bray and Dean, 2007] suggest the existence of a certain structure of critical points of random Gaussian error functions on high dimensional continuous spaces.”
On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization (2018, background): “…minima, saddles) in the landscape of the training objective, either for linear networks (see for example Kawaguchi (2016); Hardt & Ma (2016) or Baldi & Hornik (1989) for a classic account), or for speciﬁc non-linear networks under different restrictive assumptions ( cf. Choromanska et al.…”
Marginalized Denoising Autoencoders for Domain Adaptation (2012, background): “Various forms of au-toencoders have been developed in the deep learning literature (Rumelhart et al., 1986; Baldi & Hornik, 1989; Kavukcuoglu et al., 2009; Lee et al., 2009; Vincent et al., 2008; Rifai et al., 2011).”
Deep learning in neural networks: An overview (2014, background): “Many papers of the previous millennium, however, were about unsupervised learning (UL) without a teacher (e.g., Hebb, 1949; von der Malsburg, 1973; Kohonen, 1972, 1982, 1988; Willshaw and von der Malsburg, 1976; Grossberg, 1976a,b; Watanabe, 1985; Rumelhart and Zipser, 1986; Pearlmutter and Hinton, ”
A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, background): “It is known that AEs with linear activations that minimize the mean quadratic error learn the principal components of the data [42].”
Deep Learning without Poor Local Minima (2016, methodology): “This is indeed the expression of a global minimum (Baldi & Hornik, 1989; Baldi & Lu, 2012).”
Relational autoencoder for feature extraction (2017, background): “In 1989, Baldi and Hornik [34] investigated the relationship between a one-layer autoencoder and principal component analysis (PCA).”
Entropy-SGD: biasing gradient descent into wide valleys (2016, neutral): “They also relate to earlier studies such as Baldi & Hornik (1989); Fyodorov & Williams (2007); Bray & Dean (2007) which show that critical points with high training error are exponentially likely to be saddle points with many negative directions and all local minima are likely to have error that is…”
The Ideal Continual Learner: An Agent That Never Forgets (2023, background): “Under certain conditions, matrix factorization (10) is equivalent to two-layer linear neural networks, with C being the first layer of weight parameters and U the second (Baldi & Hornik, 1989; Vidal, 2020; Peng & Risteski, 2022).”
A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks (2018, background): “Baldi and Hornik (1989)) and recent (e.g. Choromanska et al. (2015); Kawaguchi (2016); Hardt and Ma (2016); Soudry and Carmon (2016); Haeffele and Vidal (2017); Nguyen and Hein (2017); Safran and Shamir (2018); Nguyen and Hein (2018); Laurent and Brecht (2018)), have focused on the validity of these”
Identity Matters in Deep Learning (2016, background): “(1.1) To give some intuition, when the depth ` is large enough, we can hope that the target function R has a factored representation in which each matrix Ai has small norm.”
Learning One-hidden-layer Neural Networks with Landscape Design (2017, methodology): “The optimization landscape has also been analyzed for other machine learning problems, including SVD/PCA phase retrieval/synchronization, orthogonal tensor decomposition, dictionary learning, matrix completion, matrix sensing [BH89, SJ13, GHJY15, SQW15, BBV16, GLM16, BNS16, GJZ17].”
An empirical analysis of the optimization of deep network loss surfaces (2016, background): “Some have characterized the critical points of deep linear neural networks [2, 28] and high-dimensional, random Gaussian error functions [4, 9, 11, 24].”
Local minima in training of neural networks (2016, background): “These intuitions can also be traced back to the earlier work of Baldi & Hornik (1989), which shows that an MLP with a single linear intermediate layer has no local minima, only saddle points and a global minimum.”
Step Size Matters in Deep Learning (2018, background): “Unless f is a quadratic function of the parameters, the system described by (1) is either a nonlinear system or a hybrid system that switches from one dynamics to another over time.”
Semantic Autoencoder for Zero-Shot Learning (2017, background): “Autoencoder There are many variants of autoencoders in the literature [5, 27, 34, 59, 46, 51].”