On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks

By Sifan Wang, Hanwen Wang, Paris Perdikaris. arXiv.org, 2020

779 citations recorded by the source corpus.

Physics-informed neural networks (PINNs) are demonstrating remarkable promise in integrating physical models with gappy and noisy observational data, but they still struggle in cases where the target functions to be approximated exhibit high-frequency or multi-scale features. In this work we investigate this limitation through the lens of Neural Tangent Kernel (NTK) theory and elucidate how PINNs are biased towards learning functions along the dominant eigen-directions of their limiting NTK. Using this observation, we construct novel architectures that employ spatio-temporal and multi-scale random Fourier features, and justify how such coordinate embedding layers can lead to robust and accurate PINN models. Numerical examples are presented for several challenging cases where conventional PINN models fail, including wave propagation and reaction-diffusion dynamics, illustrating how the proposed methods can be used to effectively tackle both forward and inverse problems involving partial differential equations with multi-scale behavior. All code an data accompanying this manuscript will be made publicly available at \url{https://github.com/PredictiveIntelligenceLab/MultiscalePINNs}.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/846be0b9ba8dd1b8e4264340a1f8b4dfc3e42cfa

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Finite basis physics-informed neural networks (FBPINNs): a scalable domain decomposition approach for solving differential equations (2021, methodology): “Similarly, spectral bias is a major issue when solving problems with multi-scale solutions [Wang et al., 2020c].”
Transformer for Partial Differential Equations' Operator Learning (2022, background): “The random Fourier projection γ ( · ) alleviate the spectral bias of coordinate-based neural networks (Tancik et al., 2020; Mildenhall et al., 2020), which has also been introduced into physics-informed machine learning in Wang et al. (2021c).”
Exact imposition of boundary conditions with distance functions in physics-informed deep neural networks (2021, background): “This example serves to reveal the spectral (low-frequency) bias [97, 98] of neural network approximations.”
CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method (2021, background): “Note that although a “tanh” activation function is most widely used for PINNs, it was suggested by recent studies that learning in the sinusoidal space of PINNs can achieve a more accurate solution [36,37].”
Characterizing possible failure modes in physics-informed neural networks (2021, neutral): “limit) to study their convergence [36, 37].”