The Dynamics of Learning: A Random Matrix Approach

By Zhenyu A. Liao, Romain Couillet. 2018

44 citations recorded by the source corpus.

Understanding the learning dynamics of neural networks is one of the key issues for the improvement of optimization algorithms as well as for the theoretical comprehension of why deep neu-ral nets work so well today. In this paper, we introduce a random matrix-based framework to analyze the learning dynamics of a single-layer linear network on a binary classification problem, for data of simultaneously large dimension and size, trained by gradient descent. Our results provide rich insights into common questions in neural nets, such as overfitting, early stopping and the initialization of training, thereby opening the door for future studies of more elaborate structures and models appearing in today's neural networks.

Subfields: Artificial Intelligence, Statistics and Probability, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Scaling & Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, background): “This phenomenon shares some similarity with the norm divergence that occurs in linear networks with mean square loss for which ||fN ||μ ∼ |N − P |−2 [27, 28].”
A jamming transition from under- to over-parametrization affects generalization in deep learning (2018, background): “Further theoretical studies on regression showed a precise mathematical description of the cusp behaviour in [30, 31, 32], albeit on models that are practically somewhat further away from modern neural networks.”
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning (2018, background): “different Heavy-Tailed Universality classes [46, 32, 18, 25, 20, 5, 111, 7, 38, 17, 90, 8, 99]; other work on RMT approaches [133, 120, 114, 112, 87, 87, 137, 85, 126]; other work on statistical physics approaches [132, 52, 117, 125, 137, 119, 113]; work on fitting to noisy versus reliable signal [1”
On the Learning Dynamics of Deep Neural Networks (2018, background): “It implies linear separability of the data, an assumption often necessary in theoretical studies (Soudry et al., 2017; Liao & Couillet, 2018; Nacson et al., 2018; Xu et al., 2018) and the positioning of the origin between the two sets.”
A jamming transition from under- to over-parametrization affects loss landscape and generalization (2018, background): “The presence of the cusp is reminiscent of the teacher-student problems, where a cusp appears either because of noise in the teacher [36, 12, 1] or because of a mean-square error loss [28].”