On the Similarity between the Laplace and Neural Tangent Kernels

By Amnon Geifman, Abhay Kumar Yadav, Yoni Kasten, Meirav Galun, David W. Jacobs, Ronen Basri. 2020

116 citations recorded by the source corpus.

Recent theoretical work has shown that massively overparameterized neural networks are equivalent to kernel regressors that use Neural Tangent Kernels(NTK). Experiments show that these kernel methods perform similarly to real neural networks. Here we show that NTK for fully connected networks is closely related to the standard Laplace kernel. We show theoretically that for normalized data on the hypersphere both kernels have the same eigenfunctions and their eigenvalues decay polynomially at the same rate, implying that their Reproducing Kernel Hilbert Spaces (RKHS) include the same sets of functions. This means that both kernels give rise to classes of functions with the same smoothness properties. The two kernels differ for data off the hypersphere, but experiments indicate that when data is properly normalized these differences are not significant. Finally, we provide experiments on real data comparing NTK and the Laplace kernel, along with a larger class of{\gamma}-exponential kernels. We show that these perform almost identically. Our results suggest that much insight about neural networks can be obtained from analysis of the well-known Laplace kernel, which has a simple closed-form.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting (2021, background): “Studies on NTK eigenvalues have made remarkable progress, covering shallow and deep ReLU neural networks (Geifman et al., 2020; Chen & Xu, 2020), skip connections (Belfer et al., 2021), and CNNs (Favero et al., 2021).”
Locality defeats the curse of dimensionality in convolutional teacher–student scenarios (2021, background): “Indeed, Θ and the Laplacian kernel share the same nonanalytic behaviour in the origin, and their spectra have the same asymptotic decay [32].”
Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHS (2020, methodology): “[17] applied the exponential power kernel and the NTK to classification and regression tasks on the UCI dataset and other large scale datasets.”
Mean-field Behaviour of Neural Tangent Kernel for Deep Neural Networks (2019, methodology): “Many concurrent results have observed this fact (Geifman et al., 2020; Cao et al., 2020; Bietti and Mairal, 2019).”
Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks (2021, methodology): “ , where λ k are the eigenvalues of r and Our characterization of the RKHS structure of ResNTK yields similar results to those shown for FC-NTK and for the Laplace kernel (Bietti & Bach, 2020; Chen & Xu, 2020; Geifman et al., 2020), yielding the following theorem.”