5分で分かる!? 有名論文ナナメ読み：Jacot, Arthor, Gabriel, Franck and Hongler, Clement : Neural Tangent Kernel : Convergence and Generalization in Neural Networks

By 俊一甘利. 2020

178 citations recorded by the source corpus.

Subfields: Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Scaling & Learning Theory

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, methodology): “5 Asymptotic generalization as N →∞ We now study the fluctuations of f t N throughout training for large networks using the Neural Tangent Kernel [22] or NTK.”
Feature Learning in Infinite-Width Neural Networks (2020, background): “For example, in NTP, r = 1/2, while in µP, r = 0.”
When do neural networks outperform kernel methods? (2020, background): “A number of recent papers show that, if weights and SGD updates are suitably scaled, and the network is sufficiently wide (N sufficiently large), then SGD converges to a function f̂NN that is approximately in F RF(W ) + F NT(W ), with W determined by the SGD initialization [18, 13, 12, 3, 34, 25].”
The Ideal Continual Learner: An Agent That Never Forgets (2023, background): “Studying continual regression is of interest as (continual) regression is closely related to deep (continual) learning in the neural tangent kernel regime (Jacot et al., 2018).”
FedBN: Federated Learning on Non-IID Features via Local Batch Normalization (2021, methodology): “So far, the categories of non-iid data considered according to Kairouz et al. (2019); Hsieh et al.”
Backward Feature Correction: How Deep Learning Performs Deep Learning (2020, background): “There is a rich literature approximating the learning process of over-parameterized networks using the neural tangent kernel (NTK) approach, where the kernel is defined by the gradient of a neural network at random initialization [35].”
LQF: Linear Quadratic Fine-Tuning (2020, background): “Work on the Neural Tangent Kernel (NTK) showed that linear models approximate the dynamics of randomly initialized deep networks [19, 28], at least when the number of filters goes to infinity, although [36] argue the analysis is valid for finite networks in the case of fine-tuning when the weights w”
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models (2021, methodology): “Our analysis for sparse model convergence relies heavily on recent advance in neural tangent kernel (NTK) [Jacot et al., 2018].”
How Do Vision Transformers Work? (2022, background): “Neural tangent kernel (NTK) (Jacot et al., 2018) theoretically implies that the loss landscape of a ViT is convex and flat when the number of heads or the number of embedding dimensions per head goes to infinity (Hron et al., 2020; Liu et al., 2020).”
On the geometry of generalization and memorization in deep neural networks (2021, background): “Previous work (Jacot et al., 2018; Lee et al., 2019) suggests that very wide networks remain near initialization where our analysis in Claim 3 holds, so we hypothesize that wider networks rely more on shared features, though this may not always be optimal for accuracy (Tsipras et al.”
A type of generalization error induced by initialization in deep neural networks (2019, methodology): “Equivalent optimization problems in the NTK regime For the completeness, we introduce the optimization problems equivalent to the training of DNNs in NTK regime, which appears explicitly or implicitly in previous works (Chizat and Bach, 2018; Oymak and Soltanolkotabi, 2018; Jacot et al., 2018; Mei e”