Understanding and mitigating gradient pathologies in physics-informed neural networks

By Sifan Wang, Yujun Teng, Paris Perdikaris. 2020

361 citations recorded by the source corpus.

The widespread use of neural networks across different scientific domains often involves constraining them to satisfy certain symmetries, conservation laws, or other domain knowledge. Such constraints are often imposed as soft penalties during model training and effectively act as domain-specific regularizers of the empirical risk loss. Physics-informed neural networks is an example of this philosophy in which the outputs of deep neural networks are constrained to approximately satisfy a given set of partial differential equations. In this work we review recent advances in scientific machine learning with a specific focus on the effectiveness of physics-informed neural networks in predicting outcomes of physical systems and discovering hidden physics from noisy data. We will also identify and analyze a fundamental mode of failure of such approaches that is related to numerical stiffness leading to unbalanced back-propagated gradients during model training. To address this limitation we present a learning rate annealing algorithm that utilizes gradient statistics during model training to balance the interplay between different terms in composite loss functions. We also propose a novel neural network architecture that is more resilient to such gradient pathologies. Taken together, our developments provide new insights into the training of constrained neural networks and consistently improve the predictive accuracy of physics-informed neural networks by a factor of 50-100x across a range of problems in computational physics. All code and data accompanying this manuscript are publicly available at \url{this https URL}.

Subfields: Aerospace Engineering, Statistical and Nonlinear Physics, Optimization & Training, AI for Physical Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Statistical and Nonlinear Physics

In Optimization & Training

In AI for Physical Sciences

Citations in the corpus

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “Wang et al. (2020a) discussed the problem of numerical stiffness in gradient backpropagation during training of a PINN.”
Physics informed deep learning for computational elastodynamics without labeled data (2020, background): “A detailed study on this issue can be found in [55].”
Finite basis physics-informed neural networks (FBPINNs): a scalable domain decomposition approach for solving differential equations (2021, background): “For example, the PINN loss function can be highly non-convex and result in a stiff optimisation problem, yet it is unclear which classes of problems this affects [Wang et al., 2020a,b].”
Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism (2020, methodology): “2e-4, which improves upon the learning-rate annealing weighted scheme proposed in [12], and begins to encroach on the accuracy of their improved fully-connected scheme with no additional modifications to the network structure itself.”
Physics-informed neural networks (PINNs) for fluid mechanics: a review (2021, methodology): “, using multi-fidelity framework [34], adaptive activation functions [35] and dynamic weights of the loss function [36], hard constraints [37] and CNN-based network architectures [38], which can improve the performance of PINNs on different problems.”
hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition (2020, background): “They may be user-speciﬁed or tuned manually or automatically, e.g., in practice based on the numerical experiment in each problem; their optimal bound, however, is still an open problem in the literature [27].”
Universal Differential Equations for Scientific Machine Learning (2020, background): “[16] Sifan Wang, Yujun Teng, and Paris Perdikaris.”
Characterizing possible failure modes in physics-informed neural networks (2021, background): “It has been argued that some of these problems may be due an imbalance in back-propagated gradients in the loss function during training, and a learning-rate annealing scheme has been proposed to mitigate this [35].”
NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations (2020, methodology): “To tackle this problem, we apply the strategy of dynamic weights [25] for choosing α and β in NSFnet simulations.”
On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks (2020, methodology): “Recent work by Wang et al. [29, 6] has identiﬁed two fundamental weaknesses in conventional PINN formulations.”
Stiff-PINN: Physics-Informed Neural Network for Stiff Chemical Kinetics (2020, background): “[21] investigated a fundamental mode of failure of PINN that is related to numerical stiffness leading to unbalanced back-propagated gradients between the loss function of initial/boundary conditions and the loss function of residuals of the differential equations during model training.”
Physics-informed graph neural Galerkin networks: A unified framework for solving PDE-governed forward and inverse problems (2021, background): “Properly tuning the penalty weight λ is critical to the convergence, which is, however, challenging and often conducted empirically [16].”
Locally adaptive activation functions with slope recovery for deep and physics-informed neural networks (2020, background): “W F and W u are the weights for residual and training data points, respectively, which can be chosen dynamically [14].”