On the Spectral Bias of Neural Networks

By Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Dräxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, Aaron C. Courville. 2018

2,472 citations recorded by the source corpus.

Neural networks are known to be a class of highly expressive functions able to fit even random input-output mappings with $100\%$ accuracy. In this work, we present properties of neural networks that complement this aspect of expressivity. By using tools from Fourier analysis, we show that deep ReLU networks are biased towards low frequency functions, meaning that they cannot have local fluctuations without affecting their global behavior. Intuitively, this property is in line with the observation that over-parameterized networks find simple patterns that generalize across data samples. We also investigate how the shape of the data manifold affects expressivity by showing evidence that learning high frequencies gets \emph{easier} with increasing manifold complexity, and present a theoretical understanding of this behavior. Finally, we study the robustness of the frequency components with respect to parameter perturbation, to develop the intuition that the parameters must be finely tuned to express high frequency functions.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yoshua Bengio
  2. Sergey Levine
  3. Ruslan Salakhutdinov
  4. Jianfeng Gao
  5. Mohit Bansal
  6. Quoc V. Le
  7. Song-Chun Zhu
  8. Percy Liang

In Statistical and Nonlinear Physics

  1. G. Em Karniadakis
  2. Lu Lu
  3. Zhiping Mao
  4. Xuhui Meng
  5. Paris Perdikaris
  6. Yoshua Bengio
  7. Andrew Markham
  8. T. Nissen‐Meyer

In Scaling & Learning Theory

  1. Zhi-Qin John Xu
  2. Yaoyu Zhang
  3. Greg Yang
  4. Aaron C. Courville
  5. Geoffrey E. Hinton
  6. Hadi Salman
  7. Meirav Galun
  8. Yan Xiao

Citations in the corpus