Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks
By Abdulkadir Canatar, Blake Bordelon, Cengiz Pehlevan. 2020
257 citations recorded by the source corpus.
A theoretical understanding of generalization remains an open problem for many machine learning models, including deep networks where overparameterization leads to better performance, contradicting the conventional wisdom from classical statistics. Here, we investigate generalization error for kernel regression, which, besides being a popular machine learning method, also describes certain infinitely overparameterized neural networks. We use techniques from statistical mechanics to derive an analytical expression for generalization error applicable to any kernel and data distribution. We present applications of our theory to real and synthetic datasets, and for many kernels including those that arise from training deep networks in the infinite-width limit. We elucidate an inductive bias of kernel regression to explain data with simple functions, characterize whether a kernel is compatible with a learning task, and show that more data may impair generalization when noisy or not expressible by the kernel, leading to non-monotonic learning curves with possibly many peaks. Canatar et al. propose a predictive theory of generalization in kernel regression applicable to real data. This theory explains various generalization phenomena observed in wide neural networks, which admit a kernel limit and generalize well despite being overparameterized.
Subfields: Artificial Intelligence, Materials Chemistry
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Shotaro Akaho
- Alessandro Favero
- Francesco Cagnetta
- Matthieu Wyart
- Ryo Karakida
- Greg Yang
- Karen Livescu
- Dacheng Tao
In Materials Chemistry
- Shotaro Akaho
- Alessandro Favero
- Francesco Cagnetta
- Matthieu Wyart
- Ryo Karakida
- Giorgio Parisi
- Francesco Zamponi
- Pierfrancesco Urbani
Citations in the corpus
- Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting (2021, methodology): “Its implementation is also given by Canatar et al. (2021).”
- Locality defeats the curse of dimensionality in convolutional teacher–student scenarios (2021, methodology): “In this appendix we provide additional details about the derivation of equation (8) within the framework of [17, 18].”