Characterizing possible failure modes in physics-informed neural networks

By Aditi S. Krishnapriyan, A. Gholami, Shandian Zhe, Robert M. Kirby, Michael W. Mahoney. Neural Information Processing Systems, 2021

1,222 citations recorded by the source corpus.

Recent work in scientific machine learning has developed so-called physics-informed neural network (PINN) models. The typical approach is to incorporate physical domain knowledge as soft constraints on an empirical loss function and use existing machine learning methodologies to train the model. We demonstrate that, while existing PINN methodologies can learn good models for relatively trivial problems, they can easily fail to learn relevant physical phenomena for even slightly more complex problems. In particular, we analyze several distinct situations of widespread physical interest, including learning differential equations with convection, reaction, and diffusion operators. We provide evidence that the soft regularization in PINNs, which involves PDE-based differential operators, can introduce a number of subtle problems, including making the problem more ill-conditioned. Importantly, we show that these possible failure modes are not due to the lack of expressivity in the NN architecture, but that the PINN's setup makes the loss landscape very hard to optimize. We then describe two promising solutions to address these failure modes. The first approach is to use curriculum regularization, where the PINN's loss term starts from a simple PDE regularization, and becomes progressively more complex as the NN gets trained. The second approach is to pose the problem as a sequence-to-sequence learning task, rather than learning to predict the entire space-time at once. Extensive testing shows that we can achieve up to 1-2 orders of magnitude lower error with these methods as compared to regular PINN training.

Subfields: Aerospace Engineering, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/3c4372b125d0744bb68bfca9f5d6b0abb85dd182

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Statistical and Nonlinear Physics

Citations in the corpus

Earthformer: Exploring Space-Time Transformers for Earth System Forecasting (2022, background): “Recent studies on adding physical constraints [20, 26] and ensembling the predictions from a data-driven model and a physics-based model [32] imply that it is an active and promising research direction to pursue.”
A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks (2022, positive): “For time-dependent problems, it is usually helpful to ﬁrst train PINNs within a short time domain and then gradually expand the time intervals of training until the entire time domain is covered [22, 23, 24, 25, 26].”
PDEBENCH: An Extensive Benchmark for Scientific Machine Learning (2022, methodology): “Here, we consider a one-dimensional diffusion-reaction type PDE, that combines a diffusion process and a rapid evolution from a source term [27].”
Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next (2022, background): “In general, PINN can fail to approximate a solution, not due to the lack of expressivity in the NN architecture but due to soft PDE constraint optimization problems Krishnapriyan et al (2021).”
Integrating Scientific Knowledge with Machine Learning for Engineering and Environmental Systems (2020, neutral): “5) [226] [251] [301] [303] [70] [196] [228] [248] [318] [82] [102] [144] [292] [216] [195] [23] [90] [320] [154] [58] [236] [53] [70] [192] [252] [26] [61] [149] [88] [299] [216] [195] [225] [145] [198] [171] [44] [188]”
Transformer for Partial Differential Equations' Operator Learning (2022, background): “[35], for certain problems such as reaction-diffusion equation, this could lead to poor performance.”