Frequency Bias in Neural Networks for Input of Non-Uniform Density

By Ronen Basri, Meirav Galun, Amnon Geifman, David W. Jacobs, Yoni Kasten, Shira Kritchman. 2020

249 citations recorded by the source corpus.

Recent works have partly attributed the generalization ability of over-parameterized neural networks to frequency bias -- networks trained with gradient descent on data drawn from a uniform distribution find a low frequency fit before high frequency ones. As realistic training sets are not drawn from a uniform distribution, we here use the Neural Tangent Kernel (NTK) model to explore the effect of variable density on training dynamics. Our results, which combine analytic and empirical observations, show that when learning a pure harmonic function of frequency $\kappa$, convergence at a point $\x \in \Sphere^{d-1}$ occurs in time $O(\kappa^d/p(\x))$ where $p(\x)$ denotes the local density at $\x$. Specifically, for data in $\Sphere^1$ we analytically derive the eigenfunctions of the kernel associated with the NTK for two-layer networks. We further prove convergence results for deep, fully connected networks with respect to the spectral decomposition of the NTK. Our empirical study highlights similarities and differences between deep and shallow networks in this model.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Model Architectures

In Optimization & Training

Citations in the corpus

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (2020, methodology): “[16] and extended by others [2, 3, 20] shows that when the width of the layers in f tends to inﬁnity and the learning rate for SGD tends to zero, the function f ( x ; θ ) converges over the course of training to the kernel regression solution using the neural tangent kernel (NTK), deﬁned as: When…”
Unsupervised part representation by Flow Capsules (2020, methodology): “This can be attributed to the architecture we use for mask decoders, and the inductive bias of MLPs in generating low-frequency functions (Tancik et al., 2020; Atzmon & Lipman, 2020; Basri et al., 2020; Rahaman et al., 2019).”
Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks (2021, methodology): “ , where λ k are the eigenvalues of r and Our characterization of the RKHS structure of ResNTK yields similar results to those shown for FC-NTK and for the Laplace kernel (Bietti & Bach, 2020; Chen & Xu, 2020; Geifman et al., 2020), yielding the following theorem.”
Hyena Hierarchy: Towards Larger Convolutional Language Models (2023, background): “This allows (7) to learn filters with high-frequency content, addressing the low-frequency bias of neural networks (Basri et al., 2020).”
When and why PINNs fail to train: A neural tangent kernel perspective (2020, background): “The term “spectral bias" [29, 41, 32] refers to a well known pathology that prevents deep fully-connected networks from learning high-frequency functions.”
On the Similarity between the Laplace and Neural Tangent Kernels (2020, background): “[7] Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman.”
COTR: Correspondence Transformer for Matching Across Images (2021, methodology): “To achieve this, we leverage the inductive bias that densely connected networks possess in representing smooth functions [1, 4, 48] and use a transformer [73, 10, 14]”