Understanding training and generalization in deep learning by Fourier analysis

By Zhi-Qin John Xu. 2018

115 citations recorded by the source corpus.

Background: It is still an open research area to theoretically understand why Deep Neural Networks (DNNs)---equipped with many more parameters than training data and trained by (stochastic) gradient-based methods---often achieve remarkably low generalization error. Contribution: We study DNN training by Fourier analysis. Our theoretical framework explains: i) DNN with (stochastic) gradient-based methods often endows low-frequency components of the target function with a higher priority during the training; ii) Small initialization leads to good generalization ability of DNN while preserving the DNN's ability to fit any function. These results are further confirmed by experiments of DNNs fitting the following datasets, that is, natural images, one-dimensional functions and MNIST dataset.

Subfields: Artificial Intelligence, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks (2020, neutral): “A line of work adopted the Fourier perspective and demonstrated that low-frequency functions are often learned first [Rahaman et al., 2018, Xu, 2018, Xu et al., 2019a,b].”
On the Spectral Bias of Neural Networks (2018, background): “This generalizes the argument made in Xu (2018) for two layer sigmoid networks by observing that the gradient w.r.t parameters of the network function inherits the spectral decay rate of the function itself 10 .”
Towards Understanding the Spectral Bias of Deep Learning (2019, background): “[Xu, 2018] provided Fourier analysis to two-layer networks and showed similar empirical results on one-dimensional functions and real data.”
A Fine-Grained Spectral Perspective on Neural Networks (2019, background): “Several other works also adopt a spectral perspective on neural networks [12, 54, 18, 7, 64, 72, 65, 66]; here we highlight a few most relevant to us.”
Training behavior of deep neural network in frequency domain (2018, methodology): “The theoretical framework [22,21] of analyzing the F-Principle is used to analyze a nonlinear collaborative scheme for deep network training [27].”
SGD on Neural Networks Learns Functions of Increasing Complexity (2019, background): “Several prior works propose measures of the complexity of neural networks, and claim that training involves learning simple patterns [4, 40, 35, 33].”
Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks (2019, background): “5 ) has a bit lower prediction accuracy.”
Explaining Knowledge Distillation by Quantifying the Knowledge (2020, methodology): “The Fourier analysis [44] was applied to explain the generalization of DNNs learned by stochastic gradient descent.”
A type of generalization error induced by initialization in deep neural networks (2019, background): “Previous works (Xu et al. 2018, Xu 2018a,b, Xu et al. 2019, Rahaman et al. 2018) discover a Frequency-Principle (F-Principle) that DNNs prefer to learn the training data by a low-frequency function.”