On the Inductive Bias of Neural Tangent Kernels

By Alberto Bietti, Julien Mairal. 2019

316 citations recorded by the source corpus.

State-of-the-art neural networks are heavily over-parameterized, making the optimization algorithm a crucial ingredient for learning predictive models with good generalization properties. A recent line of work has shown that in a certain over-parameterized regime, the learning dynamics of gradient descent are governed by a certain kernel obtained at initialization, called the neural tangent kernel. We study the inductive bias of learning in such a regime by analyzing this kernel and the corresponding function space (RKHS). In particular, we study smoothness, approximation, and stability properties of functions with finite norm, including stability to image deformations in the case of convolutional networks, and compare to other known kernels for similar architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Statistical Learning Theory

Citations in the corpus

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (2020, methodology): “In this setting, the NTK is a function of the dot product between its inputs and of their norms [3, 5, 6, 16].”
Linear Mode Connectivity in Multitask and Continual Learning (2020, background): “…linearizations seem to hold reliably in over parametrized models throughout learning as shown by the NTK-based literature (e.g Jacot et al., 2018; Bietti & Mairal, 2019) and these assumptions are implicitly made by several successful continual learning algorithms (e.g. Kirkpatrick et al., 2017;…”
A Fine-Grained Spectral Perspective on Neural Networks (2019, background): “Finally, we note that several recent [64] or concurrent [9, 8] works studied one-hidden layer ReLU neural networks over the sphere, building on Smola et al.”
Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks (2019, methodology): “Consider the ReLU random feature kernel and let H K denote the RKHS associated with this kernel.”
Frequency Bias in Neural Networks for Input of Non-Uniform Density (2020, methodology): “(Bach, 2017; Basri et al., 2019; Bietti & Mairal, 2019; Cao et al., 2019; Xie et al., 2017) For a deep FC network the NTK is expressed by the following recursion (Arora et al., 2019a; Jacot et al., 2018) Here ˙ σ ( · ) denotes the step function (i.e., the derivative of the ReLU function).”
Locality defeats the curse of dimensionality in convolutional teacher–student scenarios (2021, methodology): “In [8] the effects of the architecture’s locality are studied from a kernel perspective, using a class of deep convolutional kernels introduced in [26, 27] and characterising their reproducing kernel Hilbert space (RKHS).”
The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies (2019, methodology): “[4], building on the work of [3], perform a frequency analysis of the inductive bias of networks, using the Neural Tangent Kernel.”
A Theoretical Analysis of Deep Q-Learning (2019, background): “; Zou et al. (2018); Chizat et al. (2019); Allen-Zhu et al. (2019a,b); Jacot et al. (2018); Cao and Gu (2019); Arora et al. (2019); Weinan et al. (2019); Mei et al. (2019); Yehudai and Shamir (2019); Bietti and Mairal (2019); Yang and Salman (2019); Yang (2019); Gao et al. (2019); Bai and Lee (2019)”
Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks (2020, background): “Neural networks are thought to generalize well because of implicit regularization 2,51,52 .”
Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks (2020, background): “In addition to training dynamics, recent works such as (Yang & Salman, 2019; Bietti & Mairal, 2019; Cao et al., 2019) have discussed how the spectrum of kernels impacts its smoothness and approximation properties.”
Convergence of Adversarial Training in Overparametrized Neural Networks (2019, unknown): “We have the following lemma to characterize the capacity of the RKHS. Lemma C.3 (RKHS Contains Smooth Functions, Proposition 2 in [4], Corollary 6 in [7]) .”
Mean-field Behaviour of Neural Tangent Kernel for Deep Neural Networks (2019, methodology): “The spectral decomposition of zonal kernels on the sphere is a classical result in spectral theory which was recently applied to Neural Tangent Kernel Geifman et al. (2020); Cao et al. (2020); Bietti and Mairal (2019).”
On the infinite width limit of neural networks with a standard parameterization (2020, background): “Inﬁnite width Bayesian [21, 15, 17, 18, 23, 9, 7, 28, 29, 6] and gradient descent trained [12, 16, 5, 28, 13, 8, 3, 1, 2, 26] neural networks are an area of active and extremely promising work.”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, background): “Among them, [12, 106, 18, 22] decompose the learning process along the principal components of the NTK.”
On the Similarity between the Laplace and Neural Tangent Kernels (2020, methodology): “[5, 11] further leveraged the spectral properties of the kernels to investigate their RKHS in the case of bias-free two layer networks.”
Finite Depth and Width Corrections to the Neural Tangent Kernel (2019, background): “A number of articles [3, 8, 15, 22] have followed up on the original NTK work [14].”
Towards Understanding the Spectral Bias of Deep Learning (2019, methodology): “Based on the technique in [Bach, 2017], [Bietti and Mairal, 2019] studied the eigenvalue decay of integrating operator defined by the neural tangent kernel on unit sphere by using spherical harmonics.”