Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems

By Maizar Raissi, Paris Perdikaris, G. Em Karniadakis. 2018

284 citations recorded by the source corpus.

The process of transforming observed data into predictive mathematical models of the physical world has always been paramount in science and engineering. Although data is currently being collected at an ever-increasing pace, devising meaningful models out of such observations in an automated fashion still remains an open problem. In this work, we put forth a machine learning approach for identifying nonlinear dynamical systems from data. Specifically, we blend classical tools from numerical analysis, namely the multi-step time-stepping schemes, with powerful nonlinear function approximators, namely deep neural networks, to distill the mechanisms that govern the evolution of a given data-set. We test the effectiveness of our approach for several benchmark problems involving the identification of complex, nonlinear and chaotic dynamics, and we demonstrate how this allows us to accurately learn the dynamics, forecast future states, and identify basins of attraction. In particular, we study the Lorenz system, the fluid flow behind a cylinder, the Hopf bifurcation, and the Glycoltic oscillator model as an example of complicated nonlinear dynamics typical of biological systems.

Subfields: Global and Planetary Change, Statistical and Nonlinear Physics, Numerical Methods & Scientific Computing, AI for Physical Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Global and Planetary Change

In Statistical and Nonlinear Physics

In Numerical Methods & Scientific Computing

In AI for Physical Sciences

Citations in the corpus

Neural Ordinary Differential Equations (2018, neutral): “One can train feed-forward or recurrent neural networks to approximate a differential equation (Raissi and Karniadakis, 2018; Raissi et al., 2018a; Long et al., 2017), with applications such as ﬂ uid simulation (Wiewel et al., 2018).”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “…have been used in many scientific applications, including fluid mechanics modeling (Raissi et al., 2020), cardiac activation mapping (Sahli Costabal et al., 2020), stochastic systems modeling (Yang and Perdikaris, 2019), and discovery of differential equations (Raissi et al., 2018; Raissi, 2018).”
Understanding and mitigating gradient pathologies in physics-informed neural networks (2020, background): “Examples of cases include recent studies on physics-informed neural networks [12, 44], which primarily use fully-connected architectures with weak inductive biases and simply rely on the fact that the representational capacity of these over-parametrized networks can be sufficient to correctly captur”
Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations (2018, background): “However, in many cases of practical interest this choice still remains an art and systematic studies with respect to the neural network architectures often fail to reveal consistent patterns [34, 35, 4].”
Adversarial Uncertainty Quantification in Physics-Informed Neural Networks (2018, background): “[6, 7, 8, 9, 10] set foot exactly at this relatively unexplored interface between applied mathematics and contemporary machine learning by revisiting the idea of penalizing the loss function of deep neural networks using diﬀerential equation constraints, as ﬁrst put forth by Psichogios and Ungar…”
Neural Jump Stochastic Differential Equations (2019, background): “More broadly, learning parameters of differential equations from data has been successful for physics-based problems with deterministic dynamics [33, 34, 35, 36, 37].”
Machine Learning of Space-Fractional Differential Equations (2018, methodology): “The methodology therein was further utilized in [30] and [23] to train neural networks to distill the actual dynamics of nonlinear dynamical systems and nonlinear partial differential equations, respectively.”
Neural Granger Causality (2021, background): “Indeed, their time series variants, such as autoregressive multilayer perceptrons (MLPs) [23], [24], [25] and recurrent neural networks (RNNs) like long-short term memory networks (LSTMs) [26] have shown impressive performance in forecasting multivariate time series given their past [27], [28],…”
Forward-Backward Stochastic Neural Networks: Deep Learning of High-dimensional Partial Differential Equations (2018, methodology): “In addition, modern methods for solving forward and inverse problems associated with partial diﬀerential equations approximate the unknown solution u ( t, x ) by either a neural network [4–6 , 20 ] or a Gaussian process [ 7–”
Universal Differential Equations for Scientific Machine Learning (2020, methodology): “While work has demonstrated that efficiency is greatly improved when incorporating classical discretization techniques into the training process (as “discrete physics-informed neural networks”, such as in multi-step neural networks [17]), the formalism provided by PINNs is not conducive to the full ”
Data Driven Governing Equations Approximation Using Deep Neural Networks (2018, background): “The topic has been explored in a series of recent articles, in the context of ODEs [36, 39]) and PDEs ([34, 33, 27]).”
Hidden Fluid Mechanics: A Navier-Stokes Informed Deep Learning Framework for Assimilating Flow Visualization Data (2018, methodology): “Lack of training data on c for t < 0 and t > 16 leads to weaker neural network predictions for the initial and ﬁnal time instants.”
Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators (2019, methodology): “The network architectures they employed includes FNNs [24], recurrent neural networks (RNNs) [20], reservoir computing [20], residual networks [23], autoencoder [9], neural ordinary diﬀerential equations [6], and neural jump stochastic diﬀerential equations [12].”
Granger Causality: A Review and Recent Advances (2021, background): “Indeed, their time series variants, such as autoregressive multilayer perceptrons (MLPs) (Raissi et al., 2018; Ki¸si, 2004; Billings, 2013) and recurrent neural networks (RNNs) like long-short term memory networks (LSTMs) (Graves, 2012) have shown impressive performance in forecasting multivariate…”
Solving Allen-Cahn and Cahn-Hilliard Equations using the Adaptive Physics Informed Neural Networks (2020, background): “In the past few years, there has been intensive research on understanding how deep neural networks can be adopted to solve and discover differential equations [1–3, 7, 8, 15, 17, 20–22, 24, 28].”
Data-driven discovery of coordinates and governing equations (2019, unknown): “In addition to methods for discovering linear embeddings [18, 33], deep learning has also been used for parameter estimation of PDEs [36, 37].”