Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks
By Grant M. Rotskoff, Eric Vanden-Eijnden. 2018
144 citations recorded by the source corpus.
Subfields: Optimization & Training, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Optimization & Training
- Greg Yang
- Sam Schoenholz
- Jeffrey Pennington
- Lechao Xiao
- Jaehoon Lee
- Jascha Narain Sohl-Dickstein
- Boris Hanin
- Roman Novak
In Scaling & Learning Theory
- Greg Yang
- Sam Schoenholz
- Lechao Xiao
- Jaehoon Lee
- Jeffrey Pennington
- Boris Hanin
- Roman Novak
- Jascha Narain Sohl-Dickstein
Citations in the corpus
- Wide neural networks of any depth evolve as linear models under gradient descent (2019, background): “[23], Chizat and Bach [24], Rotskoff and Vanden-Eijnden [25], Sirignano and Spiliopoulos [26] analyze the mean field SGD dynamics of training neural networks in the large-width limit.”
- The large learning rate phase of deep learning: the catapult mechanism (2020, background): “…limit of infinite width (Daniely, 2017; Jacot et al., 2018; Lee et al., 2019; Du et al., 2019; Zou et al., 2018; Allen-Zhu et al., 2019; Li & Liang, 2018; Chizat et al., 2019; Mei et al., 2018; Rotskoff & Vanden-Eijnden, 2018; Sirignano & Spiliopoulos, 2018; Woodworth et al., 2019; Naveh et al.).”
- Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, background): “The complementary “mean-field” limit of finite input dimension and an infinite number of hidden neurons was analysed (Mei et al., 2018; Chizat & Bach, 2018; Sirignano & Spiliopoulos, 2020; Rotskoff & Vanden-Eijnden, 2018).”
- Dynamics of stochastic gradient descent for two-layer neural networks in the teacher–student setup (2019, unknown): “We also note that several recent theorems [29, 31, 30] about the global convergence of SGD do not apply in our setting because we have a finite number of hidden units.”
- A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics (2019, background): “For this reason, there has been a lot of recent theoretical work centered on these issues [15, 16, 12, 11, 2, 8, 10, 31, 29, 28, 25, 27].”
- Finite Depth and Width Corrections to the Neural Tangent Kernel (2019, background): “Another interpretation is that the specific NTK scaling of weights at initialization [4, 5, 17, 18, 19, 20] and the implicit small learning rate limit [16] obscure important aspects of SGD dynamics.”