Mean field analysis of neural networks: A central limit theorem

By Justin A. Sirignano, Konstantinos Spiliopoulos. 2018

218 citations recorded by the source corpus.

We rigorously prove a central limit theorem for neural network models with a single hidden layer. The central limit theorem is proven in the asymptotic regime of simultaneously (A) large numbers of hidden units and (B) large numbers of stochastic gradient descent training iterations. Our result describes the neural network's fluctuations around its mean-field limit. The fluctuations have a Gaussian distribution and satisfy a stochastic partial differential equation. The proof relies upon weak convergence methods from stochastic analysis. In particular, we prove relative compactness for the sequence of processes and uniqueness of the limiting process in a suitable Sobolev space.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Statistics and Probability, Scaling & Learning Theory, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Statistics and Probability

In Scaling & Learning Theory

In Statistical Learning Theory

Citations in the corpus

On Lazy Training in Differentiable Programming (2018, background): “In contrast, the critical scaling α(m) = 1/m, allows to converge as m→∞ to a non degenerate dynamic described by a partial differential equation and referred to as the mean-field limit [24, 7, 29, 33].”
Mean Field Analysis of Neural Networks: A Law of Large Numbers (2018, background): “In related work which builds upon the results in this paper, a central limit theorem has been proven for single-layer neural networks in [46] and a law of large numbers has been proven for deep neural networks in [47].”
Mean Field Analysis of Deep Neural Networks (2019, background): “This has motivated recent mathematical research on deep learning models such as [22], [23], [24], [25], [26], [21], [31], [32], [27], and [30].”
Dynamics of stochastic gradient descent for two-layer neural networks in the teacher–student setup (2019, unknown): “[32] J.”
Toward Moderate Overparameterization: Global Convergence Guarantees for Training Shallow Neural Networks (2019, result): “We would also like to note that a few interesting recent papers [29,38–40] relate the empirical distribution of the network parameters to Wasserstein gradient flows using ideas from mean field analysis.”
Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss (2020, background): “, 2006; Bach, 2017a) or training behavior (Nitanda and Suzuki, 2017; Rotskoff and Vanden-Eijnden, 2018; Chizat and Bach, 2018; Mei et al., 2018; Sirignano and Spiliopoulos, 2019), which can be described by a Wasserstein gradient flow (Ambrosio et al.”
A Note on Lazy Training in Supervised Differentiable Programming (2018, background): “Note that this scaling contrasts with the scaling α(m) = 1/m chosen in a series of recent works that study the mean-field limit of neural networks with a single hidden layer [23, 6, 26, 28].”
A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics (2019, background): “For this reason, there has been a lot of recent theoretical work centered on these issues [15, 16, 12, 11, 2, 8, 10, 31, 29, 28, 25, 27].”
A mean-field limit for certain deep neural networks (2019, result): “However, Sirignano and Spiliopoulos note that there are difficulties with this and related ideas.”
A Generalization Theory of Gradient Descent for Learning Over-parameterized Deep ReLU Networks (2019, background): “Mei et al. (2018); Chizat and Bach (2018); Sirignano and Spiliopoulos (2018); Rotskoff and Vanden-Eijnden (2018); Wei et al. (2018) took a distributional view of over-parametrized networks, used mean field analysis to show that the empirical distribution of the two-layer neural network parameters…”