On-line learning in soft committee machines.

By David Saad, Sara A. Solla. 1995

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, background): “In the perceptron, the simplest network without hidden layers, the cusp in the test error at the jamming point is also observed and predicted analytically [50, 51, 52, 53, 54, 55].”
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization (2014, background): “In (Saad and Solla, 1995) the dynamics of stochastic gradient descent are analyzed for soft committee machines.”
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, methodology): “Parameters: M = 2 , K = 1 , V = 1 . classical work on standard supervised learning by Saad & Solla (1995a); Riegler & Biehl (1995).”
On Lazy Training in Differentiable Programming (2018, background): “As seen on panel (c), this behavior hinders good generalization in the teacher-student setting [30].”
Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural Networks (2017, background): “Saad and Solla [40] study the online dynamics of learning a fully connected neural network with one-hidden layer in a Gaussian planted model (or Gaussian student-teacher model).”
Escaping From Saddle Points - Online Stochastic Gradient for Tensor Decomposition (2015, background): “In general, symmetry is known to generate saddle points, and variants ofgradient descent often perform reasonably in these cases (see (Saad and Solla, 1995), (Rattray et al., 1998), (Inoue et al., 2003)).”
A jamming transition from under- to over-parametrization affects loss landscape and generalization (2018, background): “The presence of the cusp is reminiscent of the teacher-student problems, where a cusp appears either because of noise in the teacher [36, 12, 1] or because of a mean-square error loss [28].”
Data-dependence of plateau phenomenon in learning with neural network—statistical mechanical analysis (2020, unknown): “3 Evaluation of expectation terms for specific activation functions Expectation terms I2, I3 and I4 can be analytically determined for some activation functions g, including sigmoid-like g(x) = erf(x/ √ 2) (see Saad and Solla [1995]) and g(x) = ReLU(x) (see Yoshida et al.”
Statistical Mechanics of On-line Learning (2009, background): “With this specific choice, the averaging in the equations of motion (30, 31, 32) can be performed analytically for general K and M [22,23,24] Independent of the particular choice of learning algorithms a general problem in two-layered networks is caused by the inherent permutation symmetry: The i-th”
Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm (2019, methodology): “The Teacher-Student paradigm has been broadly used to study supervised learning (Saad and Solla, 1995; Monasson and Zecchina, 1995; Opper and Saad, 2001; Engel and Van den Broeck, 2001; Zdeborová and Krzakala, 2016; Barbier et al., 2019; Gabrié et al., 2018; Aubin et al., 2018; Franz et al., 2018).”
Natural Gradient Works Eciently in Learning (background): “…preliminary analysis (Yang and Amari, 1997a) by using a simple model shows that the performance of natural gradient learning is remarkably good and it is sometimes free from being trapped in plateaus which give rise to slow convergence of the backpropagation learning method (Saad and Solla, 1995).”
Adaptive Method of Realizing Natural Gradient Learning for Multilayer Perceptrons (2000, unknown): “LETTER Communicated by David Saad”
Complexity Issues in Natural Gradient Descent Method for Training Multilayer Perceptrons (1998, methodology): “When bi = 0, i = 1, · · · ,m, applying the multivariate gaussian average method in Saad and Solla (1995), we can obtain analytic expressions for these coefficients: cij, bij, and b̃ij.”
Dynamics of stochastic gradient descent for two-layer neural networks in the teacher–student setup (2019, unknown): “Our main contributions are as follows: (i) The dynamics of SGD (online) learning by two-layer neural networks in the teacher-student setup was studied in a series of classic papers [41, 42, 43, 44, 45] from the statistical physics community, leading to a heuristic derivation of a set of coupled ordi”
Efficient approaches for escaping higher order saddle points in non-convex optimization (2016, background): “Therefore they can “fool” the (ordinary) first and second order algorithms and they may stuck there for long periods of time.”
Dynamics of Learning Near Singularities in Layered Networks (2008, unknown): “A statisticalphysical approach has clarified that this phenomenon arises from symmetry (see, for example, Riegler & Biehl, 1995; Saad & Solla, 1995a, b; Biehl, Riegler, & Wöhler, 1996; Inoue, Park, & Okada, 2003, 2004; Park, Inoue, & Okada, 2003, 2005; see also Biehl & Schwarze, 1995; Biehl &…”