Mean Field Analysis of Deep Neural Networks

By Justin A. Sirignano, Konstantinos Spiliopoulos. 2019

94 citations recorded by the source corpus.

We analyze multilayer neural networks in the asymptotic regime of simultaneously (a) large network sizes and (b) large numbers of stochastic gradient descent training iterations. We rigorously establish the limiting behavior of the multilayer neural network output. The limit procedure is valid for any number of hidden layers, and it naturally also describes the limiting behavior of the training loss. The ideas that we explore are to (a) take the limits of each hidden layer sequentially and (b) characterize the evolution of parameters in terms of their initialization. The limit satisfies a system of deterministic integro-differential equations. The proof uses methods from weak convergence and stochastic analysis. We show that, under suitable assumptions on the activation functions and the behavior for large times, the limit neural network recovers a global minimum (with zero loss for the objective function).

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Statistical Learning Theory

Citations in the corpus

Mean Field Analysis of Neural Networks: A Law of Large Numbers (2018, background): “In related work which builds upon the results in this paper, a central limit theorem has been proven for single-layer neural networks in [46] and a law of large numbers has been proven for deep neural networks in [47].”
Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks (2020, methodology): “…approach is to model DNNs with nested measures (also known as multi-level measures; see Dawson et al. (1982); Dawson (2018) and references therein), which however suffers the closure problem to establish a well-deﬁned limit (see discussions in (Sirignano & Spiliopoulos, 2019b, Section 4.3)).”