Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators

By Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, G. Em Karniadakis. Nature Machine Intelligence, 2019

4,061 citations recorded by the source corpus.

It is widely known that neural networks (NNs) are universal approximators of continuous functions. However, a less known but powerful result is that a NN with a single hidden layer can accurately approximate any nonlinear continuous operator. This universal approximation theorem of operators is suggestive of the structure and potential of deep neural networks (DNNs) in learning continuous operators or complex systems from streams of scattered data. Here, we thus extend this theorem to DNNs. We design a new network with small generalization error, the deep operator network (DeepONet), which consists of a DNN for encoding the discrete input function space (branch net) and another DNN for encoding the domain of the output functions (trunk net). We demonstrate that DeepONet can learn various explicit operators, such as integrals and fractional Laplacians, as well as implicit operators that represent deterministic and stochastic differential equations. We study different formulations of the input function space and its effect on the generalization error for 16 different diverse applications. Neural networks are known as universal approximators of continuous functions, but they can also approximate any mathematical operator (mapping a function to another function), which is an important capability for complex systems such as robotics control. A new deep neural network called DeepONet can lean various mathematical operators with small generalization error.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/03547cf81db895a448c3d0283bdfa20695ed26ab

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

When and why PINNs fail to train: A neural tangent kernel perspective (2020, background): “differential equations [19], fractional differential equations [20, 21], and beyond [22, 23, 24, 25].”
Transformer for Partial Differential Equations' Operator Learning (2022, background): “The pioneering work DeepONet (Lu et al., 2021a) introduces a new data-driven modeling paradigm that aims to learn the solution operator of PDEs and provides a practical realization of the general universal nonlinear operator approximation theorem (Chen & Chen, 1995).”
KAN: Kolmogorov-Arnold Networks (2024, positive): “We refer to Deep Ritz Method [79], PINNs [22, 23] for PDE solving, and Fourier Neural operator [80], PINOs [81, 82, 83], DeepONet [84] for operator learning methods learning the solution map.”
ClimaX: A foundation model for weather and climate (2023, neutral): “These approaches are complemented by general machine learning models for fluid dynamics (Li et al., 2020; Kochkov et al., 2021; Lu et al., 2021; Brandstetter et al., 2022a;b).”
Neural Operator: Learning Maps Between Function Spaces (2021, methodology): “DeepONet Recently, a novel operator regression architecture, named DeepONet, was proposed by Lu et al. (2019, 2021) that designs a generic neural network based on the approximation theorem from Chen and Chen (1995).”
Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control (2021, background): “…exist, including equation-free modeling [1], normal form identiﬁcation [2–4], nonlinear Laplacian spectral analysis [5], Koop-man analysis [6, 7] and dynamic mode decomposition (DMD) [8–10], symbolic regression [11–15], sparse regression [16, 17], Gaussian processes [18], and deep learning [19–24].”
MIONet: Learning multiple-input operators via tensor product (2022, methodology): “The current theoretical analysis of neural operators focuses on the approximation capability such as the universal approximation theorems for DeepONet [3, 27] and FNO [17].”
Multiwavelet-based Operator Learning for Differential Equations (2021, background): “[53] Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis.”
A physics-informed variational DeepONet for predicting the crack path in brittle materials (2021, methodology): “In the ﬁrst experiment, we compare the accuracy and eﬃciency of V-DeepONet with solely data driven DeepONet[39].”
DeepXDE: A Deep Learning Library for Solving Differential Equations (2019, methodology): “In addition to solving differential equations, DeepXDE can also be used to approximate functions from multi-fidelity data [40], and learn nonlinear operators [34].”
Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next (2022, methodology): “DeepXDE is used by the authors for: for inverse scattering (Chen et al, 2020b), or deriving mechanical characteristics from materials with MFNNs (Lu et al, 2020).”
Choose a Transformer: Fourier or Galerkin (2021, background): “[90, 91] proposed a DeepONet roughly equivalent to an additive attention, similar to the one in the Neural Turing Machine (NMT) in [7].”
Neural operators for accelerating scientific simulations and design (2023, positive): “For instance, inspired by the classical Petrov-Galerkin method [30], the kernel κ ( x, y ) in (1) is constrained to be separable, yielding the DeepONet model [31].”
Learning the solution operator of parametric partial differential equations with physics-informed DeepONets (2021, positive): “Motivated by the universal approximation theorem for operators ( 35 , 36 ), the architecture features two neural networks coined as the “branch” and “trunk” networks, respectively; the automatic differentiation of which enables us to learn the solution operator of arbitrary PDEs.”