Training behavior of deep neural network in frequency domain

By Zhi-Qin John Xu, Yaoyu Zhang, Yan Xiao. 2018

413 citations recorded by the source corpus.

Why deep neural networks (DNNs) capable of overfitting often generalize well in practice is a mystery [#zhang2016understanding]. To find a potential mechanism, we focus on the study of implicit biases underlying the training process of DNNs. In this work, for both real and synthetic datasets, we empirically find that a DNN with common settings first quickly captures the dominant low-frequency components, and then relatively slowly captures the high-frequency ones. We call this phenomenon Frequency Principle (F-Principle). The F-Principle can be observed over DNNs of various structures, activation functions, and training algorithms in our experiments. We also illustrate how the F-Principle help understand the effect of early-stopping as well as the generalization of DNNs. This F-Principle potentially provides insights into a general principle underlying DNN optimization and generalization.

Subfields: Artificial Intelligence, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks (2020, neutral): “A line of work adopted the Fourier perspective and demonstrated that low-frequency functions are often learned first [Rahaman et al., 2018, Xu, 2018, Xu et al., 2019a,b].”
On the Spectral Bias of Neural Networks (2018, background): “3 A similar result has been independently found and reported in Xu et al. (2018).”
Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks (2019, background): “, 2015), and implicit bias of the training process (Neyshabur et al., 2014; Poggio et al., 2018; Soudry et al., 2018; Arpit et al., 2017; Xu et al., 2018).”
A Fine-Grained Spectral Perspective on Neural Networks (2019, background): “Several other works also adopt a spectral perspective on neural networks [12, 54, 18, 7, 64, 72, 65, 66]; here we highlight a few most relevant to us.”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, methodology): “Different decompositions of the learning process for neural networks have been used: [83, 104, 87, 105] study the learning in the Fourier domain and show that low-frequency functions are learned earlier than high-frequency ones.”
Implicit Regularization via Neural Feature Alignment (2020, result): “5 shows an alignment of the tangent kernel eigenfunctions with Fourier modes of increasing frequency, in line with prior empirical observations (Rahaman et al., 2019; Xu et al., 2019) of a ‘spectral bias’ towards low-frequency functions.”
Understanding training and generalization in deep learning by Fourier analysis (2018, methodology): “Xu et al. (2018) found similar results as the following.”
A type of generalization error induced by initialization in deep neural networks (2019, methodology): “Based on F-principle, Xu et al. (2018, 2019) point out that the final output of a DNN tends to inherit high frequencies of its initial output that can not be well constrained by the training data (Xu et al. 2018, 2019).”