Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks

By Phan-Minh Nguyen. 2019

75 citations recorded by the source corpus.

Can multilayer neural networks -- typically constructed as highly complex structures with many nonlinearly activated neurons across layers -- behave in a non-trivial way that yet simplifies away a major part of their complexities? In this work, we uncover a phenomenon in which the behavior of these complex networks -- under suitable scalings and stochastic gradient descent dynamics -- becomes independent of the number of neurons as this number grows sufficiently large. We develop a formalism in which this many-neurons limiting behavior is captured by a set of equations, thereby exposing a previously unknown operating regime of these networks. While the current pursuit is mathematically non-rigorous, it is complemented with several experiments that validate the existence of this behavior.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Scaling & Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, background): “2) [37, 38, 39] and on the asymptotic behavior of wide networks [40, 41, 42, 43].”
Feature Learning in Infinite-Width Neural Networks (2020, background): “Several works also proposed different versions of mean field frameworks for deeper MLPs (Araújo et al., 2019; Fang et al., 2020; Nguyen, 2019; Nguyen & Pham, 2020; Sirignano & Spiliopoulos, 2020).”
Disentangling feature and lazy learning in deep neural networks: an empirical study (2019, background): “…Another limit, feature learning also called mean ﬁeld in the literature, has been studied in several works focusing mostly on one-hidden layer networks [Mei et al., 2018, Rotskoff and Vanden-Eijnden, 2018, Chizat and Bach, 2018, Sirignano and Spiliopoulos, 2018, Mei et al., 2019, Nguyen, 2019].”
Mean Field Analysis of Deep Neural Networks (2019, background): “[45] also derives asymptotics for multi-layer neural networks.”
A mean-field limit for certain deep neural networks (2019, methodology): “Our work is closely related to the nonrigorous (and independent) results of Nguyen [21].”