When and why PINNs fail to train: A neural tangent kernel perspective

By Sifan Wang, Xinling Yu, Paris Perdikaris. Journal of Computational Physics, 2020

1,613 citations recorded by the source corpus.

Physics-informed neural networks (PINNs) have lately received great attention thanks to their flexibility in tackling a wide range of forward and inverse problems involving partial differential equations. However, despite their noticeable empirical success, little is known about how such constrained neural networks behave during their training via gradient descent. More importantly, even less is known about why such models sometimes fail to train at all. In this work, we aim to investigate these questions through the lens of the Neural Tangent Kernel (NTK); a kernel that captures the behavior of fully-connected neural networks in the infinite width limit during training via gradient descent. Specifically, we derive the NTK of PINNs and prove that, under appropriate conditions, it converges to a deterministic kernel that stays constant during training in the infinite-width limit. This allows us to analyze the training dynamics of PINNs through the lens of their limiting NTK and find a remarkable discrepancy in the convergence rate of the different loss components contributing to the total training error. To address this fundamental pathology, we propose a novel gradient descent algorithm that utilizes the eigenvalues of the NTK to adaptively calibrate the convergence rate of the total training error. Finally, we perform a series of numerical experiments to verify the correctness of our theory and the practical effectiveness of the proposed algorithms. The data and code accompanying this manuscript are publicly available at \url{this https URL}.

Subfields: Aerospace Engineering, Statistical and Nonlinear Physics, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/52b975170b33e19855b2970b2e978fa3ea786465

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Statistical and Nonlinear Physics

In Statistics and Probability

Citations in the corpus

Characterizing possible failure modes in physics-informed neural networks (2021, neutral): “limit) to study their convergence [36, 37].”
On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks (2020, methodology): “Recent work by Wang et al. [29, 6] has identiﬁed two fundamental weaknesses in conventional PINN formulations.”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, background): “Therefore, the common practice is to provide the theoretical analysis in simpliﬁed settings and validate the results empirically in more general cases (see, e.g. [45, 24, 101]).”
A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks (2022, methodology): “In PINNs, the total loss is a weighted summation of multiple loss terms corresponding to the PDE and initial/boundary conditions, and diﬀerent methods have been developed to automatically tune these weights and balance the losses [13, 14, 15].”
Self-adaptive loss balanced Physics-informed neural networks (2021, background): “So far, there have been several efforts to integrating sustainable, balanced learning of loss function in PINNs, such as the Neural Tangent Kernel(NTK) principle [37], the training gradients [38], and the network weights [39].”
PDEBENCH: An Extensive Benchmark for Scientific Machine Learning (2022, result): “Our baseline results further indicate that the PINNs might deal better than expected with high-frequency features, despite prior observations [62].”
Physics-Informed Neural Networks for Heat Transfer Problems (2021, background): “Figure 13 and Table 1 present comparisons of the identiﬁed parameters between the original PINN formulation of Raissi et al. [23], and the proposed PINN formulation with adaptive weights [58,59].”
Transolver: A Fast Transformer Solver for PDEs on General Geometries (2024, methodology): “Physics-informed neural networks This paradigm formalizes PDE constraints, including equations, initial and boundary conditions as objective functions of deep models (Weinan & Yu, 2017; Raissi et al., 2019; Wang et al., 2020a;b).”
Finite basis physics-informed neural networks (FBPINNs): a scalable domain decomposition approach for solving differential equations (2021, background): “Work by Wang et al. [2020b], Shin et al. [2020], Mishra and Molinaro [2020] and others have made initial steps towards understanding the theoretical properties of PINNs but this sub-ﬁeld is in its early stages.”
Physics-informed neural networks (PINNs) for fluid mechanics: a review (2021, background): “On the theoretical side, some recent works [39, 40,41] have provided more guarantees and insights into the convergence of PINNs.”
Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next (2022, methodology): “Only a few have looked into alternatives to automatic differentiation [44] or at convergence problems [179, 181].”