Dynamics of stochastic gradient descent for two-layer neural networks in the teacher–student setup

By Sebastian Goldt, Madhu S. Advani, Andrew M. Saxe, Florent Krzakala, Lenka Zdeborov'a. 2019

186 citations recorded by the source corpus.

Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data. We study this phenomenon by analysing the dynamics and the performance of over-parameterised two-layer neural networks in the teacher–student setup, where one network, the student, is trained on data generated by another network, called the teacher. We show how the dynamics of stochastic gradient descent (SGD) is captured by a set of differential equations and prove that this description is asymptotically exact in the limit of large inputs. Using this framework, we calculate the final generalisation error of student networks that have more parameters than their teachers. We find that the final generalisation error of the student increases with network size when training only the first layer, but stays constant or even decreases with size when training both layers. We show that these different behaviours have their root in the different solutions SGD finds for different activation functions. Our results indicate that achieving good generalisation in neural networks goes beyond the properties of SGD alone and depends on the interplay of at least the algorithm, the model architecture, and the data set.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, background): “…series of classic papers by Biehl & Schwarze (1995) and Saad & Solla (1995a) who derived a set of closed ODEs that track the test error of the student (see also Saad & Solla (1995b); Biehl et al. (1996); Saad (2009) for further results and Goldt et al. (2019) for a recent proof of these equations).”
Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent (2020, neutral): “Another line of works provide closed form expressions of the training dynamics of overparameterized neural networks, leveraging tools from statistical physics (Goldt et al. 2019, Goldt et al. 2020).”
If deep learning is the answer, what is the question? (2020, background): “Paradoxically, infinite-size networks can be more interpretable than those with fewer units, because their learning trajectory is stabler and not prone to being waylaid by bad local minima in the loss landscape, leading to suboptimal outcomes 141,144,146 (Fig.”
Implicit Regularization in Deep Learning May Not Be Explainable by Norms (2020, background): “5 Existing characterizations focus on different aspects of learning, for example: dynamics of optimization ([1, 25, 45, 6, 28, 41, 26]); curvature (“flatness”) of obtained minima ([52]); frequency spectrum of learned input-output mappings ([61]); invariant quantities throughout training ([24]); and ”
Data-dependence of plateau phenomenon in learning with neural network—statistical mechanical analysis (2020, unknown): “…mechanical method is applied to analyze recent techniques (Hara et al. [2016], Yoshida et al. [2017], Takagi et al. [2019] and Straat and Biehl [2019]), and generalization performance in over-parameterized setting (Goldt et al. [2019]) and environment with conceptual drift (Straat et al. [2018]).”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, background): “More recently, with the introduction of neural tangent kernel (NTK) [52, 62], a new line of research is to study the convergence properties of gradient descent [e.g. 8, 69, 25, 29, 7, 44, 33, 110, 11, 99].”
Modelling the influence of data structure on learning in neural networks (2019, background): “It was introduced by Gardner & Derrida [11] and has played a major role in theoretical studies of the generalisation ability of neural networks from an average-case perspective, particularly within the framework of statistical mechanics [3, 12, 4, 5, 13, 14, 15, 16, 17], and also in recent statistic”