On the Spectral Bias of Neural Networks
By Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Dräxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, Aaron C. Courville. 2018
2,472 citations recorded by the source corpus.
Neural networks are known to be a class of highly expressive functions able to fit even random input-output mappings with $100\%$ accuracy. In this work, we present properties of neural networks that complement this aspect of expressivity. By using tools from Fourier analysis, we show that deep ReLU networks are biased towards low frequency functions, meaning that they cannot have local fluctuations without affecting their global behavior. Intuitively, this property is in line with the observation that over-parameterized networks find simple patterns that generalize across data samples. We also investigate how the shape of the data manifold affects expressivity by showing evidence that learning high frequencies gets \emph{easier} with increasing manifold complexity, and present a theoretical understanding of this behavior. Finally, we study the robustness of the frequency components with respect to parameter perturbation, to develop the intuition that the parameters must be finely tuned to express high frequency functions.
Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Ruslan Salakhutdinov
- Jianfeng Gao
- Mohit Bansal
- Quoc V. Le
- Song-Chun Zhu
- Percy Liang
In Statistical and Nonlinear Physics
- G. Em Karniadakis
- Lu Lu
- Zhiping Mao
- Xuhui Meng
- Paris Perdikaris
- Yoshua Bengio
- Andrew Markham
- T. Nissen‐Meyer
In Scaling & Learning Theory
- Zhi-Qin John Xu
- Yaoyu Zhang
- Greg Yang
- Aaron C. Courville
- Geoffrey E. Hinton
- Hadi Salman
- Meirav Galun
- Yan Xiao
Citations in the corpus
- On the Anatomy of MCMC-based Maximum Likelihood Learning of Energy-Based Models (2019, background): “hibit much faster MCMC convergence but we find that they cannot achieve realistic synthesis. This is consistent with the claim that increase in spectral norm is needed to learn high-frequency features [30]. When Langevin noise is tuned correctly, the learning dynamics naturally balance between the ex”
- The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks (2020, neutral): “A line of work adopted the Fourier perspective and demonstrated that low-frequency functions are often learned first [Rahaman et al., 2018, Xu, 2018, Xu et al., 2019a,b].”
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (2020, background): “Analysis of the NTK’s eigendecomposition shows that its eigenvalue spectrum decays rapidly as a function of frequency, which explains the widely-observed “spectral bias” of deep networks towards learning low-frequency functions [3, 4, 33].”
- Neural Additive Models: Interpretable Machine Learning with Neural Nets (2020, background): “This difficulty of learning large local fluctuations with ReLU networks without affecting their global behavior when fitting jagged functions might be due to their bias towards smoothness [2, 32].”
- Why do tree-based models still outperform deep learning on typical tabular data? (2022, neutral): “This is in line with Rahaman et al. [2019], which finds that neural networks are biased toward low-frequency functions.”
- Solving Irregular and Data-enriched Differential Equations using Deep Neural Networks (2019, background): “The inherent spectral bias [44] of the DNNs seems to be able to contain the potential instability of the solution until the optimization has settled in a “reasonable” local minimum.”
- Drawing early-bird tickets: Towards more efficient training of deep networks (2019, neutral): “(Rahaman et al., 2019; Xu et al., 2019) argue that deep networks will first learn low-complexity (lower-frequency) functional components, before absorbing high-frequency features: the former being more robust to perturbations.”
- DeepXDE: A Deep Learning Library for Solving Differential Equations (2019, background): “Recent studies show that for function approximation, neural networks learn target functions from low to high frequencies [46, 54], but here we show that the learning mode of PINNs is different due to the existence of high-order derivatives.”
- Towards Understanding the Spectral Bias of Deep Learning (2019, methodology): “Following the settings in [Rahaman et al., 2019], we test target functions with different magnitudes in different components.”
- On Embeddings for Numerical Features in Tabular Deep Learning (2022, neutral): “While vanilla MLP is known to be a universal approximator [9, 16], in practice, due to optimization peculiarities, it has limitations in its learning capabilities [34].”
- Towards Frequency-Based Explanation for Robust CNN (2020, background): “On the other side, (Rahaman et al., 2018) shows that ReLU networks tend to learn low-frequency featuers first and then pick up the high-frequency components later.”
- Stiffness: A New Perspective on Generalization in Neural Networks (2019, background): “To explain the remarkable generalization properties of neural networks, it has been proposed (Rahaman et al., 2018) that the function family is biased towards low-frequency functions.”
- Learned Initializations for Optimizing Coordinate-Based Neural Representations (2020, background): “Recent works have shown that standard ReLU MLPs fail to adequately represent fine details in these complex low-dimensional signals due to a spectral bias [28] and address this issue by either replacing the ReLU activations with sine functions [33] or by lifting the input coordinates into a Fourier f”
- Why do tree-based models still outperform deep learning on tabular data? (2022, neutral): “This is in line with Rahaman et al. [2019], which finds that NNs are biased toward low-frequency functions.”
- Focal Frequency Loss for Image Reconstruction and Synthesis (2020, background): “Due to the inherent bias of neural networks [24, 31], a model tends to eschew some frequency components that”
- Understanding training and generalization in deep learning by Fourier analysis (2018, methodology): “2In Rahaman et al. (2018), “for matrix-valued weights, their spectral norm was computed by evaluating the eigenvalue of the eigenvector obtained with 10 power iterations.”
- NeRF: Representing scenes as neural radiance fields for view synthesis (neutral): “This leverages work in the learning community on spectral bias in neural networks [ 52 ].”
- Finite basis physics-informed neural networks (FBPINNs): a scalable domain decomposition approach for solving differential equations (2021, background): “This is the well-studied observation that neural networks tend to learn higher frequencies much more slowly than lower frequencies [Xu et al., 2019, Rahaman et al., 2019, Basri et al., 2019, Cao et al., 2019], with various convergence rates being proved.”