Mean Field Analysis of Neural Networks: A Law of Large Numbers

By Justin A. Sirignano, Konstantinos Spiliopoulos. 2018

241 citations recorded by the source corpus.

Machine learning, and in particular neural network models, have revolutionized fields such as image, text, and speech recognition. Today, many important real-world applications in these areas are driven by neural networks. There are also growing applications in engineering, robotics, medicine, and finance. Despite their immense success in practice, there is limited mathematical understanding of neural networks. This paper illustrates how neural networks can be studied via stochastic analysis, and develops approaches for addressing some of the technical challenges which arise. We analyze one-layer neural networks in the asymptotic regime of simultaneously (A) large network sizes and (B) large numbers of stochastic gradient descent training iterations. We rigorously prove that the empirical distribution of the neural network parameters converges to the solution of a nonlinear partial differential equation. This result can be considered a law of large numbers for neural networks. In addition, a consequence of our analysis is that the trained parameters of the neural network asymptotically become independent, a property which is commonly called "propagation of chaos".

Subfields: Artificial Intelligence, Finance, Statistical and Nonlinear Physics, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Finance

In Statistical and Nonlinear Physics

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

Wide neural networks of any depth evolve as linear models under gradient descent (2019, background): “Mei et al. (2018); Chizat & Bach (2018a); Rotskoff & Vanden-Eijnden (2018); Sirignano & Spiliopoulos (2018) analyze the mean ﬁeld SGD dynamics of training neural networks in the large-width limit.”
Scaling description of generalization with number of parameters in deep learning (2019, neutral): “This result explains why the generalization performance converges as N → ∞, a result previously obtained for single hidden layer neural networks using a different approach [32, 33, 34, 35].”
The large learning rate phase of deep learning: the catapult mechanism (2020, background): “…limit of infinite width (Daniely, 2017; Jacot et al., 2018; Lee et al., 2019; Du et al., 2019; Zou et al., 2018; Allen-Zhu et al., 2019; Li & Liang, 2018; Chizat et al., 2019; Mei et al., 2018; Rotskoff & Vanden-Eijnden, 2018; Sirignano & Spiliopoulos, 2018; Woodworth et al., 2019; Naveh et al.).”
Feature Learning in Infinite-Width Neural Networks (2020, methodology): “…limits (e.g. (Chizat & Bach, 2018; Fang et al., 2020; Jacot et al., 2018; Mei et al., 2018; Nguyen & Pham, 2020; Rotskoff & Vanden-Eijnden, 2018; Sirignano & Spiliopoulos, 2018)) assume various forms of regularity conditions, such as 1) 0th, 1st, and/or 2nd order smoothness on the nonlinearity…”
Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks (2019, methodology): “…ﬁeld analysis to establish that for inﬁnitely wide neural networks, the empirical distribution of the neural network parameters can be described as a Wasserstein gradient ﬂow (Mei et al., 2018; Chizat & Bach, 2018a; Sirignano & Spiliopoulos, 2018; Rotskoﬀ & Vanden-Eijnden, 2018; Wei et al., 2018).”
Surprises in High-Dimensional Ridgeless Least Squares Interpolation (2019, unknown): “The infinite width limit was also studied (beyond the linearized regime) in [18, 50, 59, 62].”
Dynamically Stable Infinite-Width Limits of Neural Classifiers (2020, methodology): “In particular, two different types of limit models have been already extensively discussed in the literature: an NTK model [1] and a mean-field limit model [2, 3, 4, 5, 6, 7].”
Disentangling feature and lazy learning in deep neural networks: an empirical study (2019, background): “…Another limit, feature learning also called mean ﬁeld in the literature, has been studied in several works focusing mostly on one-hidden layer networks [Mei et al., 2018, Rotskoff and Vanden-Eijnden, 2018, Chizat and Bach, 2018, Sirignano and Spiliopoulos, 2018, Mei et al., 2019, Nguyen, 2019].”
Trainability and Accuracy of Artificial Neural Networks: An Interacting Particle System Approach (2018, background): “This observation allows us to assert that convergence towards equilibrium of the empirical distribution occurs on a time scale that is independent of n to leading order—similar results were obtained in [12, 27, 33].”