Towards Understanding the Spectral Bias of Deep Learning

By Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuan Zhou, Quanquan Gu. 2019

313 citations recorded by the source corpus.

An intriguing phenomenon observed during training neural networks is the spectral bias, which states that neural networks are biased towards learning less complex functions. The priority of learning functions with low complexity might be at the core of explaining the generalization ability of neural networks, and certain efforts have been made to provide a theoretical explanation for spectral bias. However, there is still no satisfying theoretical result justifying the underlying mechanism of spectral bias. In this paper, we give a comprehensive and rigorous explanation for spectral bias and relate it with the neural tangent kernel function proposed in recent work. We prove that the training process of neural networks can be decomposed along different directions defined by the eigenfunctions of the neural tangent kernel, where each direction has its own convergence rate and the rate is determined by the corresponding eigenvalue. We then provide a case study when the input data is uniformly distributed over the unit sphere, and show that lower degree spherical harmonics are easier to be learned by over-parameterized neural networks. Finally, we provide numerical experiments to demonstrate the correctness of our theory. Our experimental results also show that our theory can tolerate certain model misspecification in terms of the input data distribution.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks (2020, background): “Frankle and Carbin [2019] found that sparse, trainable subnetworks – “lottery tickets" – emerge early in training.”
Benign overfitting in linear regression (2019, background): “…noise is added to the y i s.) The eigenvalues of the covariance operator in this case can have a heavy tail under reasonable assumptions on the data distribution (see [38], where this kernel was introduced, and [17]), and the dimension is very large but ﬁnite, as required for benign overﬁtting.”
Mean-field Behaviour of Neural Tangent Kernel for Deep Neural Networks (2019, methodology): “The spectral decomposition of zonal kernels on the sphere is a classical result in spectral theory which was recently applied to Neural Tangent Kernel Geifman et al. (2020); Cao et al. (2020); Bietti and Mairal (2019).”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, background): “Among them, [12, 106, 18, 22] decompose the learning process along the principal components of the NTK.”
Finite basis physics-informed neural networks (FBPINNs): a scalable domain decomposition approach for solving differential equations (2021, background): “This is the well-studied observation that neural networks tend to learn higher frequencies much more slowly than lower frequencies [Xu et al., 2019, Rahaman et al., 2019, Basri et al., 2019, Cao et al., 2019], with various convergence rates being proved.”
Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks (2021, background): “It indicates whether a kernel exhibits a frequency bias (Cao et al., 2019; Rahaman et al., 2019; Xu et al., 2019), it provides an estimate of the number of gradient descent iterations needed to learn certain target functions (Basri et al., 2019), and it can be used to estimate the generalization…”
When and why PINNs fail to train: A neural tangent kernel perspective (2020, background): “Specifically, it has been shown that conventional fullyconnected architectures – such as the ones typically used in PINNs – suffer from “spectral bias" and are incapable of learning functions with high frequencies, both in theory and in practice [29, 30, 31, 32].”
Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks (2020, background): “In addition to training dynamics, recent works such as (Yang & Salman, 2019; Bietti & Mairal, 2019; Cao et al., 2019) have discussed how the spectrum of kernels impacts its smoothness and approximation properties.”
Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks (2020, result): “In the limit where the width of a network is taken to in ﬁ nity (network is thus overparameterized), neural network training with a certain random initialization scheme can be described by ridgeless kernel regression with the Neural Network Gaussian Process kernel (NNGPK) if only the last layer is t”
On the Similarity between the Laplace and Neural Tangent Kernels (2020, background): “For two layer, FC networks, [5, 7, 14] investigated the spectral property of the NTK when the data is distributed uniformly on the hypersphere, showing in particular that with GD low frequencies are learned before higher ones.”
Frequency Bias in Neural Networks for Input of Non-Uniform Density (2020, background): “Additional work (Bach, 2017; Basri et al., 2019; Bietti & Mairal, 2019; Cao et al., 2019) proved the existence of frequency bias in NTK models of two-layer networks and derived convergence rates of training as a function of target frequency.”
On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks (2020, background): “[33, 34, 35], and its connection to investigating spectral bias [30, 31, 32] in the training behavior of deep fully-connected networks.”