Exact solution for on-line learning in multilayer neural networks.

By David Saad, Sara A. Solla. 1995

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

High-dimensional dynamics of generalization error in neural networks (2017, background): “We mainly consider a student-teacher scenario, where a “teacher” neural network generates possibly noisy samples for a “student” network to learn from (Seung, Sompolinsky, and Tishby, 1992; Watkin, Rau, and Biehl, 1993; Saad and Solla, 1995).”
The Loss Surfaces of Multilayer Networks (2014, methodology): “In the 1990s, a number of researchers studied the convergence of gradient-based learning for multilayer networks using the methods of statistical physics, i.e. [Saad and Solla, 1995], and the edited works [Saad, 2009].”
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, methodology): “Parameters: M = 2 , K = 1 , V = 1 . classical work on standard supervised learning by Saad & Solla (1995a); Riegler & Biehl (1995).”
On Natural Learning and Pruning in Multilayered Perceptrons (2000, background): “…(Rattray et al., 1998), which, using standard gradients, is the cause of long plateaus (long time spans in which the performance of the MLP barely improves—also referred to as temporary minima; see, among others, Ampazis, Perantonis, & Taylor, 1999; Saad & Solla, 1995; Wiegerinck & Heskes, 1996).”
Deep Net Triage: Assessing the Criticality of Network Layers by Structural Compression (2018, methodology): “Similar to using intermediate hints, as done in Romero et al. (2014), or Student-Teacher networks as introduced in Saad & Solla (1995), we strive to mimic the output of the parent network at the end of a compressed layer.”
Dynamics of stochastic gradient descent for two-layer neural networks in the teacher–student setup (2019, unknown): “Our main contributions are as follows: (i) The dynamics of SGD (online) learning by two-layer neural networks in the teacher-student setup was studied in a series of classic papers [41, 42, 43, 44, 45] from the statistical physics community, leading to a heuristic derivation of a set of coupled ordi”
Machine learning and the physical sciences (2019, background): “Both the stochastic gradient descent (aka online) learning (Saad and Solla, 1995a,b) and the optimal batch-learning generalization error can be analyzed in closed form in this case (Schwarze, 1993).”
The Power of Normalization: Faster Evasion of Saddle Points (2016, background): “In Dauphin et al. (2014); Sun et al. (2016), it is demonstrated how to avoid saddle points through a trust region approach which utilizes second order derivatives.”
Dynamics of Learning Near Singularities in Layered Networks (2008, unknown): “A statistical-physical approach has clarified that this phenomenon arises from symmetry (see, e.g., Riegler & Biehl, 1995; Saad & Solla, 1995a, 1995b; Biehl, Riegler, & W¨ ohler, 1996; Inoue, Park, & Okada, 2003, 2004; Park, Inoue, & Okada, 2003, 2005; see also Biehl & Schwarze, 1995; Biehl & Cati”
Statistical Mechanics of On-line Learning (2009, background): “With this specific choice, the averaging in the equations of motion (30, 31, 32) can be performed analytically for general K and M [22,23,24] Independent of the particular choice of learning algorithms a general problem in two-layered networks is caused by the inherent permutation symmetry: The i-th”