On the Risk of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels
By Tengyuan Liang, Alexander Rakhlin, Xiyu Zhai. 2019
29 citations recorded by the source corpus.
We study the risk of minimum-norm interpolants of data in a Reproducing Kernel Hilbert Space where kernel is defined as a function of the inner product. Our upper bounds on the risk are of a multiple-descent shape for the various scalings of d = n, α ∈ (0, 1). At the heart of our analysis is a study of spectral properties of the random kernel matrix restricted to a filtration of eigen-spaces of the population covariance operator. Since gradient flow on appropriately initialized wide neural networks converges to a minimum-norm interpolant, the analysis also yields estimation guarantees for these models.
Subfields: Artificial Intelligence, Computational Mechanics, Mathematical Physics, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jonathan S. Rosenfeld
- Yonatan Belinkov
- Nir Shavit
- Amir Rosenfeld
- Andrew Jones
- James R. Glass
- Nadir Durrani
- Hassan Sajjad
In Scaling & Learning Theory
Citations in the corpus
- A Constructive Prediction of the Generalization Error Across Scales (2019, background): “While fully engaging with this literature is beyond our scope, we note that recent studies have derived bounds involving power-law dependencies in both model (Yarotsky, 2018) and data size (Liang et al., 2019).”
- Triple descent and the two kinds of overfitting: where and why do they appear? (2020, unknown): “In the non-parametric setting of [34], an upper bound on the test loss is shown to exhibit multiple descent, with peaks at each N = D, i ∈ N.”