Neural Networks as Interacting Particle Systems: Asymptotic Convexity of the Loss Landscape and Universal Scaling of the Approximation Error

By Grant M. Rotskoff, Eric Vanden-Eijnden. 2018

214 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

Citations in the corpus

Wide neural networks of any depth evolve as linear models under gradient descent (2019, background): “[25] Grant M Rotskoff and Eric Vanden-Eijnden.”
Scaling description of generalization with number of parameters in deep learning (2019, neutral): “This result explains why the generalization performance converges as N → ∞, a result previously obtained for single hidden layer neural networks using a different approach [32, 33, 34, 35].”
Feature Learning in Infinite-Width Neural Networks (2020, methodology): “In summary, For all θ ∈ R, ft(ξ) stays fixed for all t and ξ if we set al ← al + θ, bl ← bl − θ, c← c− 2θ. (13) This insight in particular implies MFP is a special case of MUP in the case of 1-hidden-layer MLPs.”
When do neural networks outperform kernel methods? (2020, background): “Further even in the infinite width limit, it is known that two-layers neural networks can actually capture a richer class of functions than the associated RKHS, provided SGD training is scaled differently from the lazy regime [23, 7, 28, 30, 8].”
On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport (2018, background): “Several independent works [24, 28, 32] have studied the many-particle limit of training a neural network with a single large hidden layer and a quadratic loss R.”
Mean Field Analysis of Neural Networks: A Law of Large Numbers (2018, background): “8], and the references therein. Upon completion of this work, we became aware of the very recent work of [38] where a related PDE limit result for neural networks is derived; see also the recent work [41]. Our convergence analysis, setup, and assumptions are dierent. In [38], it is assumed that the ”
A mean-field limit for certain deep neural networks (2019, background): “) The authors of [18, 24, 25] noted that, for shallow networks, one can rewrite this evolution as”
Disentangling feature and lazy training in deep neural networks (2019, background): “A limiting behavior of neural networks, called ‘mean ﬁeld’ in the literature, has been studied in several works focusing mostly on one-hidden layer networks (Song et al 2018, Rotskoﬀ and Vanden-Eijnden 2018, Chizat and Bach 2018, Sirignano and Spiliopoulos 2018, Song et al 2019, Nguyen 2019).”
Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss (2020, background): “, 2006; Bach, 2017a) or training behavior (Nitanda and Suzuki, 2017; Rotskoff and Vanden-Eijnden, 2018; Chizat and Bach, 2018; Mei et al., 2018; Sirignano and Spiliopoulos, 2019), which can be described by a Wasserstein gradient flow (Ambrosio et al.”
Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representations (2019, background): “Recently, a different line of papers (Mei et al., 2018; Chizat & Bach, 2018; Sirignano & Spiliopoulos, 2019; Rotskoff & Vanden-Eijnden, 2018; E, 2019b; Mei et al., 2019; Dou & Liang, 2019; Wei et al., 2018) applied the mean filed view to study the over-parameterized two level NNs.”
Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks (2020, background): “(ii) Another line of research applies the mean-ﬁeld analysis for NNs (Mei et al., 2018; Chizat & Bach, 2018; Sirig-nano & Spiliopoulos, 2019a; Rotskoff & Vanden-Eijnden, 2018; Mei et al., 2019; Dou & Liang, 2019; Wei et al., 2018; Sirignano & Spiliopoulos, 2019b; Fang et al., 2019; Ara´ujo et al.,…”