Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks
By Cong Fang, Jason D. Lee, Pengkun Yang, Tong Zhang. 2020
63 citations recorded by the source corpus.
This paper proposes a new mean-field framework for over-parameterized deep neural networks (DNNs), which can be used to analyze neural network training. In this framework, a DNN is represented by probability measures and functions over its features (that is, the function values of the hidden units over the training data) in the continuous limit, instead of the neural network parameters as most existing studies have done. This new representation overcomes the degenerate situation where all the hidden units essentially have only one meaningful hidden unit in each middle layer, and further leads to a simpler representation of DNNs, for which the training objective can be reformulated as a convex optimization problem via suitable re-parameterization. Moreover, we construct a non-linear dynamics called neural feature flow, which captures the evolution of an over-parameterized DNN trained by Gradient Descent. We illustrate the framework via the standard DNN and the Residual Network (Res-Net) architectures. Furthermore, we show, for Res-Net, when the neural feature flow process converges, it reaches a global minimal solution under suitable conditions. Our analysis leads to the first global convergence proof for over-parameterized neural network training with more than $3$ layers in the mean-field regime.
Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- J. Edward Hu
- Greg Yang
- Sam Schoenholz
- Soufiane Hayou
- Jeffrey Pennington
- Arnaud Doucet
- Yuanzhi Li
- Judith Rousseau
In Statistical and Nonlinear Physics
In Scaling & Learning Theory
- J. Edward Hu
- Greg Yang
- Sam Schoenholz
- Jeffrey Pennington
- Soufiane Hayou
- Lechao Xiao
- Arnaud Doucet
- Judith Rousseau
Citations in the corpus
- Feature Learning in Infinite-Width Neural Networks (2020, methodology): “Technical Assumptions Earlier works on neural tangent or mean field limits (e.g. (Chizat & Bach, 2018; Fang et al., 2020; Jacot et al., 2018; Mei et al., 2018; Nguyen & Pham, 2020; Rotskoff & Vanden-Eijnden, 2018; Sirignano & Spiliopoulos, 2018)) assume various forms of regularity conditions, such…”