Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHS

By Lin Chen, Sheng Xu. 2020

103 citations recorded by the source corpus.

We prove that the reproducing kernel Hilbert spaces (RKHS) of a deep neural tangent kernel and the Laplace kernel include the same set of functions, when both kernels are restricted to the sphere $\mathbb{S}^{d-1}$. Additionally, we prove that the exponential power kernel with a smaller power (making the kernel more non-smooth) leads to a larger RKHS, when it is restricted to the sphere $\mathbb{S}^{d-1}$ and when it is defined on the entire $\mathbb{R}^d$.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting (2021, background): “Studies on NTK eigenvalues have made remarkable progress, covering shallow and deep ReLU neural networks (Geifman et al., 2020; Chen & Xu, 2020), skip connections (Belfer et al., 2021), and CNNs (Favero et al., 2021).”
Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks (2021, methodology): “ , where λ k are the eigenvalues of r and Our characterization of the RKHS structure of ResNTK yields similar results to those shown for FC-NTK and for the Laplace kernel (Bietti & Bach, 2020; Chen & Xu, 2020; Geifman et al., 2020), yielding the following theorem.”
On the Similarity between the Laplace and Neural Tangent Kernels (2020, result): “Indeed, the subsequent work of [16] proves thatHβ((2)) ⊇ Hβ, thus together with our results proving thatHβ((2)) = Hβ.”