A mean-field limit for certain deep neural networks

By D. Araújo, Roberto Imbuzeiro Oliveira, Daniel Yukimura. 2019

72 citations recorded by the source corpus.

Understanding deep neural networks (DNNs) is a key challenge in the theory of machine learning, with potential applications to the many fields where DNNs have been successfully used. This article presents a scaling limit for a DNN being trained by stochastic gradient descent. Our networks have a fixed (but arbitrary) number $L\geq 2$ of inner layers; $N\gg 1$ neurons per layer; full connections between layers; and fixed weights (or "random features" that are not trained) near the input and output. Our results describe the evolution of the DNN during training in the limit when $N\to +\infty$, which we relate to a mean field model of McKean-Vlasov type. Specifically, we show that network weights are approximated by certain "ideal particles" whose distribution and dependencies are described by the mean-field model. A key part of the proof is to show existence and uniqueness for our McKean-Vlasov problem, which does not seem to be amenable to existing theory. Our paper extends previous work on the $L=1$ case by Mei, Montanari and Nguyen; Rotskoff and Vanden-Eijnden; and Sirignano and Spiliopoulos. We also complement recent independent work on $L>1$ by Sirignano and Spiliopoulos (who consider a less natural scaling limit) and Nguyen (who nonrigorously derives similar results).

Subfields: Artificial Intelligence, Numerical Analysis, Statistics and Probability, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Numerical Analysis

In Statistics and Probability

In Statistical Learning Theory

Citations in the corpus

Feature Learning in Infinite-Width Neural Networks (2020, background): “We carry out Algorithm 1 for MLPs in all of our experiments.”
Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection (2020, unknown): “Firstly, we only analyze the simple twolayer networks but we believe our theory can be generalized to deep networks with refined analysis and more complex theoretical framework such as deep mean field theory (Araújo et al., 2019; Nguyen & Pham, 2020).”
Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks (2020, methodology): “Recently Araújo et al. (2019) pointed out the difficulty lied in the potential discontinuity of the conditional distribution under Wasserstein metric.”
Greedy Optimization Provably Wins the Lottery: Logarithmic Number of Winning Tickets is Enough (2020, methodology): “In comparison, the mean field analysis (Araújo et al., 2019; Mei et al., 2018) suggests that directly train a network with same size as the pruned model gives O ( n−1 ) discrepancy loss.”
Mean Field Analysis of Deep Neural Networks (2019, background): “Let us also mention that after our current paper appeared as a preprint on arXiv, [2] studied the limit of a multi-layer neural network, but under some important differences as compared to our paper.”
Dynamics of Deep Neural Networks and Neural Tangent Hierarchy (2019, methodology): “Training dynamics of neural networks in the mean field setting have been studied in (Mei et al., 2019; Song et al., 2018; Araújo et al., 2019; Nguyen, 2019; Sirignano & Spiliopoulos, 2019; Chizat & Bach, 2018).”