Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit

By Song Mei, Theodor Misiakiewicz, Andrea Montanari. 2019

328 citations recorded by the source corpus.

We consider learning two layer neural networks using stochastic gradient descent. The mean-field description of this learning dynamics approximates the evolution of the network weights by an evolution in the space of probability distributions in $R^D$ (where $D$ is the number of parameters associated to each neuron). This evolution can be defined through a partial differential equation or, equivalently, as the gradient flow in the Wasserstein space of probability distributions. Earlier work shows that (under some regularity assumptions), the mean field description is accurate as soon as the number of hidden units is much larger than the dimension $D$. In this paper we establish stronger and more general approximation guarantees. First of all, we show that the number of hidden units only needs to be larger than a quantity dependent on the regularity properties of the data, and independent of the dimensions. Next, we generalize this analysis to the case of unbounded activation functions, which was not covered by earlier bounds. We extend our results to noisy stochastic gradient descent. Finally, we show that kernel ridge regression can be recovered as a special limit of the mean field analysis.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Statistics and Probability, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Statistics and Probability

In Scaling & Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, background): “2) [37, 38, 39] and on the asymptotic behavior of wide networks [40, 41, 42, 43].”
On Lazy Training in Differentiable Programming (2018, background): “Other works in this line of research have appeared since the first version of this article was communicated, studying the optimization and statistical properties of various neural networks architectures in what we refer to as the lazy regime [5, 33, 32, 29, 46, 8, 27, 16, 4, 43].”
Finite Depth and Width Corrections to the Neural Tangent Kernel (2019, background): “Another interpretation is that the specific NTK scaling of weights at initialization [4, 5, 17, 18, 19, 20] and the implicit small learning rate limit [16] obscure important aspects of SGD dynamics.”
A mean-field limit for certain deep neural networks (2019, result): “Physicists and mathematicians have been using this perspective since the original boom of neural nets; see the books of Engel and van den Broeck [7] and Mézard and Montanari [20] for more information.”
Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks (2020, neutral): “…applies the mean-ﬁeld analysis for NNs (Mei et al., 2018; Chizat & Bach, 2018; Sirig-nano & Spiliopoulos, 2019a; Rotskoff & Vanden-Eijnden, 2018; Mei et al., 2019; Dou & Liang, 2019; Wei et al., 2018; Sirignano & Spiliopoulos, 2019b; Fang et al., 2019; Ara´ujo et al., 2019; Nguyen & Pham,…”
Disentangling feature and lazy learning in deep neural networks: an empirical study (2019, background): “It is equivalent to saying that the tangent kernel (which can always be deﬁned) evolves in time [Rotskoff and Vanden-Eijnden, 2018, Mei et al., 2019, Chizat and Bach, 2019].”
Dynamics of stochastic gradient descent for two-layer neural networks in the teacher–student setup (2019, neutral): “Both the mean-ﬁeld and the kernel regime crucially rely on having an inﬁnite number of nodes in the hidden layer, and the performance of the networks strongly depends on the detailed scaling used 38,39 .”
Generalization of Two-layer Neural Networks: An Asymptotic Viewpoint (2020, background): “Mei et al. (2018; 2019); Chizat and Bach (2018a); Rotskoﬀ and Vanden-Eijnden (2018); Sirignano and Spiliopoulos (2018) studied the mean-ﬁeld limit where the number of neurons h → ∞ and the second layer scaled by 1 /h , and established correspondence between the main-particle limit of gradient…”
On the Inductive Bias of Neural Tangent Kernels (2019, methodology): “, [15, 38, 37]), where weights move according (2)While we use gradients in our notations, we note that weak differentiability (e.”
Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection (2020, methodology): “In comparison, standard training of networks of size n by gradient descent yields a loss of O(1/n+ ") following the mean field analysis of Song et al. (2018); Mei et al. (2019), where " is usually a small term involving the loss of training infinitely wide networks; see Section 3.3 for more details.”
Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel (2018, background): “Mei et al. [43], Chizat and Bach [15], Sirignano and Spiliopoulos [64], Dou and Liang [18], Mei et al. [44] analyze a distributional view of over-parametrized networks.”
Kernel and Rich Regimes in Overparametrized Models (2019, background): “[17] Song Mei, Theodor Misiakiewicz, and Andrea Montanari.”
Gradients as Features for Deep Representation Learning (2020, background): “, 2019b) or some related ideas (Arora et al., 2019a; Li & Liang, 2018; Allen-Zhu et al., 2019a; Du et al., 2019; Allen-Zhu et al., 2019b; Cao & Gu, 2020; Mei et al., 2019) when the base network Fθ,ω is sufficiently wide and at random initialization.”