Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism

By Levi D. McClenny, Ulisses M. Braga-Neto. AAAI Spring Symposium: MLPS, 2020

764 citations recorded by the source corpus.

Physics-Informed Neural Networks (PINNs) have emerged recently as a promising application of deep neural networks to the numerical solution of nonlinear partial differential equations (PDEs). However, the original PINN algorithm is known to suffer from stability and accuracy problems in cases where the solution has sharp spatio-temporal transitions. These stiff PDEs require an unreasonably large number of collocation points to be solved accurately. It has been recognized that adaptive procedures are needed to force the neural network to fit accurately the stubborn spots in the solution of stiff PDEs. To accomplish this, previous approaches have used fixed weights hard-coded over regions of the solution deemed to be important. In this paper, we propose a fundamentally new method to train PINNs adaptively, where the adaptation weights are fully trainable, so the neural network learns by itself which regions of the solution are difficult and is forced to focus on them, which is reminiscent of soft multiplicative-mask attention mechanism used in computer vision. The basic idea behind these Self-Adaptive PINNs is to make the weights increase where the corresponding loss is higher, which is accomplished by training the network to simultaneously minimize the losses and maximize the weights, i.e., to find a saddle point in the cost surface. We show that this is formally equivalent to solving a PDE-constrained optimization problem using a penalty-based method, though in a way where the monotonically-nondecreasing penalty coefficients are trainable. Numerical experiments with an Allen-Cahn stiff PDE, the Self-Adaptive PINN outperformed other state-of-the-art PINN algorithms in L2 error by a wide margin, while using a smaller number of training epochs. An Appendix contains additional results with Burger's and Helmholtz PDEs, which confirmed the trends observed in the Allen-Cahn experiments.

Subfields: Artificial Intelligence, Control and Systems Engineering, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/f084bcbe162f16b1e2c83e1dfa9ff02c4fc7f7c0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Statistical and Nonlinear Physics

Citations in the corpus

Learning the solution operator of parametric partial differential equations with physics-informed DeepONets (2021, neutral): “It has been shown that these weights play an important role in enhancing the trainability of constrained neural networks ( 62 – 64 ).”
Physics‐Informed Neural Networks (PINNs) for Wave Propagation and Full Waveform Inversions (2021, methodology): “…better constrain the factors that control the convergence of PINNs and the criteria for choosing the minimum density of the training data for the PDE to guarantee convergence to the correct solution (Shin et al., 2020; Wang et al., 2020); see also the recent work of McClenny and Braga-Neto (2020).”
A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks (2022, background): “Moreover, a different weight for each loss term could be set at every training point [16, 17, 8, 18].”
Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems (2021, background): “[13, 14] show that a properly-designed non-uniform training point weighting can improve the accuracy.”
Physics-Informed Neural Operator for Learning Partial Differential Equations (2021, positive): “We add a comparison experiment against the Locally adaptive activation functions for PINN (LAAF-PINN) [54] and Self-Adaptive PINN ( SA-PINN ) [55].”
Self-adaptive loss balanced Physics-informed neural networks (2021, methodology): “So far, there have been several efforts to integrating sustainable, balanced learning of loss function in PINNs, such as the Neural Tangent Kernel(NTK) principle [37], the training gradients [38], and the network weights [39].”
CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method (2021, background): “In general, a huge amount of collocation points is required for matching the differential equations in order to train a good PINN model [42,43].”