A Fine-Grained Spectral Perspective on Neural Networks

By Greg Yang, Hadi Salman. 2019

122 citations recorded by the source corpus.

Are neural networks biased toward simple functions? Does depth always help learn more complex features? Is training the last layer of a network as good as training all layers? How to set the range for learning rate tuning? These questions seem unrelated at face value, but in this work we give all of them a common treatment from the spectral perspective. We will study the spectra of the *Conjugate Kernel, CK,* (also called the *Neural Network-Gaussian Process Kernel*), and the *Neural Tangent Kernel, NTK*. Roughly, the CK and the NTK tell us respectively "what a network looks like at initialization" and "what a network looks like during and after training." Their spectra then encode valuable information about the initial distribution and the training and generalization properties of neural networks. By analyzing the eigenvalues, we lend novel insights into the questions put forth at the beginning, and we verify these insights by extensive experiments of neural networks. We derive fast algorithms for computing the spectra of CK and NTK when the data is uniformly distributed over the boolean cube, and show this spectra is the same in high dimensions when data is drawn from isotropic Gaussian or uniformly over the sphere. Code replicating our results is available at this http URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Scaling & Learning Theory

In Statistical Learning Theory

Citations in the corpus

The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks (2020, neutral): “For multi-layer FC networks, it was known that their infinite-width NTKs have the form K(x,x′) = h( 2 d , ‖x′‖2 d , 〈x,x′〉 d ) (x,x ′ ∈ R) for some function h : R(3) → R [Yang and Salman, 2019].”
Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (2020, background): “[2], which show that the outputs of a network throughout gradient descent remain close to those of a linear dynamical system whose convergence rate is governed by the eigenvalues of the NTK matrix [2, 3, 5, 20, 43].”
Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes (2019, background): “Additionally, the NN-GP correspondence has led to new understanding of neural network training and generalization [42, 53, 61].”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, methodology): “Dominant directions in the feature space as well as the parameter space are given by principal components of the NTRF matrix Φ0, which are the same as those of the NTK Gram matrix [106].”
Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHS (2020, background): “Yang and Salman [27] analyzed the spectra of the conjugate kernel (CK) and NTK on the boolean cube.”
On the Similarity between the Laplace and Neural Tangent Kernels (2020, methodology): “[46] analyzed the eigenvalues of NTK over the Boolean cube, and [20] analyzed its spectrum under approximate pairwise orthogonality.”
Implicit Regularization via Neural Feature Alignment (2020, background): “In linearized regimes where deep learning reduces to kernel regression (Jacot et al., 2018; Du et al., 2019; Allen-Zhu et al., 2019), one can dwell further the nature of such a bias by analyzing the eigenfunctions of the neural tangent kernel (e.g., Yang & Salman, 2019).”
Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks (2020, methodology): “Neural networks are thought to generalize well because of implicit regularization 2,51,52 .”
Tensor Programs II: Neural Tangent Kernel for Any Architecture (2020, background): “For example, its spectrum informs us the kind of functions that can be learned quickly and generalize well [65].”
A Theoretical Analysis of Deep Q-Learning (2019, background): “The theory of neural tangent kernel can be extended to feedforward neural networks with multiple layers and neural networks with more complicated architectures (Gao et al., 2019; Frei et al., 2019; Yang and Salman, 2019; Yang, 2019; Huang et al., 2020).”