Universal Differential Equations for Scientific Machine Learning

By Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, K. Zubov, R. Supekar, Dominic J. Skinner, A. Ramadhan. arXiv.org, 2020

823 citations recorded by the source corpus.

In the context of science, the well-known adage “a picture is worth a thousand words” might well be “a model is worth a thousand datasets.” Scientific models, such as Newtonian physics or biological gene regulatory networks, are human-driven simplifications of complex phenomena that serve as surrogates for the countless experiments that validated the models. Recently, machine learning has been able to overcome the inaccuracies of approximate modeling by directly learning the entire set of nonlinear interactions from data. However, without any predetermined structure from the scientific basis behind the problem, machine learning approaches are flexible but data-expensive, requiring large databases of homogeneous labeled training data. A central challenge is reconciling data that is at odds with simplified models without requiring "big data". In this work demonstrate how a mathematical object, which we denote universal differential equations (UDEs), can be utilized as a theoretical underpinning to a diverse array of problems in scientific machine learning to yield efficient algorithms and generalized approaches. The UDE model augments scientific models with machine-learnable structures for scientifically-based learning. We show how UDEs can be utilized to discover previously unknown governing equations, accurately extrapolate beyond the original data, and accelerate model simulation, all in a time and data-efficient manner. This advance is coupled with open-source software that allows for training UDEs which incorporate physical constraints, delayed interactions, implicitly-defined events, and intrinsic stochasticity in the model. Our examples show how a diverse set of computationally-difficult modeling issues across scientific disciplines, from automatically discovering biological mechanisms to accelerating the training of physics-informed neural networks and large-eddy simulations, can all be transformed into UDE training problems that are efficiently solved by a single software methodology.

Subfields: Statistical and Nonlinear Physics, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/696b388ee6221c6dbcfd647a06883b2bfee773d9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistical and Nonlinear Physics

In Statistics and Probability

Citations in the corpus

Monarch: Expressive Structured Matrices for Efficient and Accurate Training (2022, methodology): “Deep learning has found application in the domain of differential equations and scientific computing [85], with methods developed for prediction and control problems [56, 71], as well as acceleration of numerical schemes [83, 51].”
Continuous-in-Depth Neural Networks (2020, background): “Along these lines, several physics-based models [66, 53, 63, 54, 17, 2] and continuous analogues to NNs [23, 14, 3] have recently been proposed.”
Characterizing possible failure modes in physics-informed neural networks (2021, methodology): “A number of tools and methodologies now exist to solve scientiﬁc problems by combining ML and domain insights [14, 20, 27, 28, 38].”
Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next (2022, methodology): “Raissi et al (2019) used different typologies of DNN, for each problem, like a 5-layer deep neural network with 100 neurons per layer, an DNN with 4 hidden layers and 200 neurons per layer or a 9 layers with 20 neuron each layer.”
Machine learning and deep learning—A review for ecologists (2022, background): “…with traditional mechanistic models (Rammer & Seidl, 2019; Reichstein et al., 2019 ), for approximating differential equations (Chen et al., 2019 ; Rackauckas et al., 2021 ), or for species distribution models (Chen et al., 2018 ; Elith & Leathwick, 2009; Harris et al., 2018 ; Wilkinson et al.,…”
Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism (2020, background): “The neural network architecture is fully connected with layer sizes [2, 100, 100, 100, 100, 1].”
Discovering Symbolic Models from Deep Learning with Inductive Biases (2020, background): “We also note several exciting packages for symbolic regression of partial differential equations on gridded data: [35, 36, 37, 38, 39, 40].”
The challenge and opportunity of battery lifetime prediction from field data (2021, background): “To this end, promising frameworks exist, including the field inversion and machine learning paradigm [105], physics-informed neural networks [106–108], neural ordinary differential equations [109], universal differential equations [110], and neural differential equations [111].”