Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

By Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yan Xiao, Zheng Ma. 2019

776 citations recorded by the source corpus.

We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective. We demonstrate a very universal Frequency Principle (F-Principle) --- DNNs often fit target functions from low to high frequencies --- on high-dimensional benchmark datasets such as MNIST/CIFAR10 and deep neural networks such as VGG16. This F-Principle of DNNs is opposite to the behavior of most conventional iterative numerical schemes (e.g., Jacobi method), which exhibit faster convergence for higher frequencies for various scientific computing problems. With a simple theory, we illustrate that this F-Principle results from the regularity of the commonly used activation functions. The F-Principle implies an implicit bias that DNNs tend to fit training data by a low-frequency function. This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or randomized dataset.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks (2020, neutral): “A line of work adopted the Fourier perspective and demonstrated that low-frequency functions are often learned first [Rahaman et al., 2018, Xu, 2018, Xu et al., 2019a,b].”
Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher (2020, background): “It is a well-known result that neural network first learns the low frequency components of the target function [24].”
Drawing early-bird tickets: Towards more efficient training of deep networks (2019, neutral): “(Rahaman et al., 2019; Xu et al., 2019) argue that deep networks will first learn low-complexity (lower-frequency) functional components, before absorbing high-frequency features: the former being more robust to perturbations.”
Adaptive activation functions accelerate convergence in deep and physics-informed neural networks (2019, methodology): “e network to learn complex functions by allowing it to capture the high frequencies components in the solution. Another such tool is the frequency principle or F-principle proposed by Xu, et al., see [36] and the references therein. In this case, the evolution of solution is observed in frequency do”
A Fine-Grained Spectral Perspective on Neural Networks (2019, background): “Several other works also adopt a spectral perspective on neural networks [13, 55, 19, 7, 66, 74, 67, 68]; here we highlight a few most relevant to us.”
The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies (2019, methodology): “A related analysis for shallow networks is presented in [25, 24].”
Towards Understanding the Spectral Bias of Deep Learning (2019, result): “Similar observation has also been pointed out in [Xu et al., 2019; Xu et al., 2020].”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, methodology): “Different decompositions of the learning process for neural networks have been used: [83, 104, 87, 105] study the learning in the Fourier domain and show that low-frequency functions are learned earlier than high-frequency ones.”
A type of generalization error induced by initialization in deep neural networks (2019, methodology): “Our study focuses on the regression problem, which plays a key role in many applications, e.g., simulation of physical systems (Zhang et al. 2018), prediction of time series (Qiu et al. 2014) and solving differential equations (E & Yu 2018, Xu et al. 2019).”
Focal Frequency Loss for Image Reconstruction and Synthesis (2020, background): “Due to the inherent bias of neural networks [24, 31], a model tends to eschew some frequency components that”
Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks (2020, background): “Other studies have shown that neural networks fit the low frequency components of the target before the high frequency components during training with gradient descent (Xu et al., 2019b; Rahaman et al., 2019; Zhang et al., 2019; Luo et al., 2019).”
Frequency Bias in Neural Networks for Input of Non-Uniform Density (2020, background): “In search of an explanation, a number of recent papers (Far-nia et al., 2018; Rahaman et al., 2019; Xu et al., 2019) have suggested that training with gradient descent (GD) (as well as SGD) yields a frequency bias – in early epochs training a neural net yields a low frequency ﬁt to the target…”
Training behavior of deep neural network in frequency domain (2018, methodology): “The theoretical framework [22,21] of analyzing the F-Principle is used to analyze a nonlinear collaborative scheme for deep network training [27].”
Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next (2022, background): “This could be an extension of the Frequency-principle, F-principle [198], according to which DNNs fit target functions from low to high frequencies during training, implying a low-frequency bias in DNNs and explaining why DNNs do not generalize well on randomized datasets.”
Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks (2021, background): “It indicates whether a kernel exhibits a frequency bias (Cao et al., 2019; Rahaman et al., 2019; Xu et al., 2019), it provides an estimate of the number of gradient descent iterations needed to learn certain target functions (Basri et al., 2019), and it can be used to estimate the generalization…”
When and why PINNs fail to train: A neural tangent kernel perspective (2020, background): “The term “spectral bias" [29, 41, 32] refers to a well known pathology that prevents deep fullyconnected networks from learning high-frequency functions.”
DeepXDE: A Deep Learning Library for Solving Differential Equations (2019, background): “Recent studies show that for function approximation, neural networks learn target functions from low to high frequencies [29, 35], but here we show that the learning mode of PINNs is different due to the existence of high-order derivatives.”
Finite basis physics-informed neural networks (FBPINNs): a scalable domain decomposition approach for solving differential equations (2021, background): “This is the well-studied observation that neural networks tend to learn higher frequencies much more slowly than lower frequencies [40–43], with various convergence rates being proved.”