Asymptotics of Wide Networks from Feynman Diagrams

By Ethan Dyer, Guy Gur-Ari. 2019

127 citations recorded by the source corpus.

Understanding the asymptotic behavior of wide networks is of considerable interest. In this work, we present a general method for analyzing this large width behavior. The method is an adaptation of Feynman diagrams, a standard tool for computing multivariate Gaussian integrals. We apply our method to study training dynamics, improving existing bounds and deriving new results on wide network evolution during stochastic gradient descent. Going beyond the strict large width limit, we present closed-form expressions for higher-order terms governing wide network training, and test these predictions empirically.

Subfields: Artificial Intelligence, Mathematical Physics, Statistics and Probability, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Mathematical Physics

In Statistics and Probability

In Scaling & Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, methodology): “Finally, our analysis is based on a scaling estimate of the fluctuations of the NTK at initialization, recently supported by more detailed analysis based on Feynman diagrams and path numbering [40, 41].”
The large learning rate phase of deep learning: the catapult mechanism (2020, methodology): “…parameters (Lee et al., 2019); this becomes exact in the inﬁnite width limit, where these dynamics are sometimes called lazy training (Jacot et al., 2018; Lee et al., 2019; Du et al., 2019; Li & Liang, 2018; Zou et al., 2018; Allen-Zhu et al., 2019; Chizat et al., 2019; Dyer & Gur-Ari, 2020).”
On the infinite width limit of neural networks with a standard parameterization (2020, background): “Inﬁnite width Bayesian [21, 15, 17, 18, 23, 9, 7, 28, 29, 6] and gradient descent trained [12, 16, 5, 28, 13, 8, 3, 1, 2, 26] neural networks are an area of active and extremely promising work.”
Explaining neural scaling laws (2021, methodology): “In this section we consider training and test data drawn uniformly from a compact d -dimensional manifold, x ∈ M d , and targets given by some smooth function y = F ( x ) on this manifold.”
Dataset Distillation with Infinitely Wide Convolutional Networks (2021, methodology): “…1994, Lee et al., 2018, Matthews et al., 2018, Jacot et al., 2018, Novak et al., 2019, Garriga-Alonso et al., 2019, Arora et al., 2019, Yang, 2019a,b, Hron et al., 2020] and the extended correspondence with the finite width corrections [Dyer and Gur-Ari, 2020, Huang and Yau, 2020, Yaida, 2020].”
Disentangling feature and lazy training in deep neural networks (2019, background): “It is clear from equation (21 Concerning the product W ℓ ˙ z ℓ , in the large-width limit it can be proven to be correctly estimated by considering that W ℓ and ˙ z ℓ are independent (Dyer and Gur-Ari 2019).”
Finite Depth and Width Corrections to the Neural Tangent Kernel (2019, methodology): “Both statements are consistent with prior results in the literature [8, 14].”