Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks

By Blake Bordelon, Abdulkadir Canatar, Cengiz Pehlevan. 2020

269 citations recorded by the source corpus.

We derive analytical expressions for the generalization performance of kernel regression as a function of the number of training samples using theoretical methods from Gaussian processes and statistical physics. Our expressions apply to wide neural networks due to an equivalence between training them and kernel regression with the Neural Tangent Kernel (NTK). By computing the decomposition of the total generalization error due to different spectral components of the kernel, we identify a new spectral principle: as the size of the training set grows, kernel machines and neural networks fit successively higher spectral modes of the target function. When data are sampled from a uniform distribution on a high-dimensional hypersphere, dot product kernels, including NTK, exhibit learning stages where different frequency modes of the target function are learned. We verify our theory with simulations on synthetic data and MNIST dataset.

Subfields: Artificial Intelligence, Scaling & Learning Theory, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Scaling & Learning Theory

In Statistical Learning Theory

Citations in the corpus

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (2020, background): “In this setting, the NTK is a function of the dot product between its inputs and of their norms [3, 5, 6, 16].”
Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting (2021, methodology): “The deﬁnitions of constants ( N ( d, i ) , c and w j ) are given in Section 8 of Bordelon et al. (2020) and we set r = 1000 as is the same in this previous work.”
Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks (2020, methodology): “Therefore, k-th eigenvalue has degeneracy of the degree k spherical harmonics, OD(D), (K,L→ k, l and ρ→ m) [31], allowing recourse to the same scaling we used to analyze rotation invariant kernels in the previous section.”
Explaining neural scaling laws (2021, methodology): “In the particular case of large width, deep neural networks behave as random feature models [12–17], and known results on the loss scaling of kernel methods can be applied [18, 19].”
Towards Understanding the Spectral Bias of Deep Learning (2019, methodology): “Very recently, [Bordelon et al., 2020] gave a spectral analysis on the generalization error of NTK-based kernel ridge regression.”
Locality defeats the curse of dimensionality in convolutional teacher–student scenarios (2021, methodology): “[17] Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan.”
Capturing the learning curves of generic features maps for realistic data sets with a teacher-student model (2021, methodology): “For the particular case of kernel ridge regression, Theorem 2 provides a rigorous proof of the formula conjectured in [37].”
Asymptotic learning curves of kernel methods: empirical data versus teacher–student paradigm (2020, methodology): “More recently, [11] generalized this approach using similar”
On the Similarity between the Laplace and Neural Tangent Kernels (2020, methodology): “NTK methods can be analyzed to explain many properties of neural networks in this limit, including their convergence in training and ability to generalize [8, 9, 13, 32].”