Transient dynamics of on-line learning in two-layered neural networks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, background): “…series of classic papers by Biehl & Schwarze (1995) and Saad & Solla (1995a) who derived a set of closed ODEs that track the test error of the student (see also Saad & Solla (1995b); Biehl et al. (1996); Saad (2009) for further results and Goldt et al. (2019) for a recent proof of these equations).”
Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural Networks (2017, methodology): “[7] use tools from statistical physics (and in particular the aforementioned dynamical equations) to demonstrate that the dynamics of training such neural networks has several fixed points and find learning-rate dependent phenomena, such as splitting and disappearing of fixed points.”
Modelling the influence of data structure on learning in neural networks (2019, background): “These “plateaus” are well-known in the literature for both SGD, where they appear as a function of time [35, 22, 36], and in batch learning, where they appear as a function of the training set size [37, 4].”