Neural-net-induced Gaussian process regression for function approximation and PDE solution

By Guofei Pang, Liu Yang, G. Em Karniadakis. 2018

80 citations recorded by the source corpus.

Abstract Neural-net-induced Gaussian process (NNGP) regression inherits both the high expressivity of deep neural networks (deep NNs) as well as the uncertainty quantification property of Gaussian processes (GPs). We generalize the current NNGP to first include a larger number of hyperparameters and subsequently train the model by maximum likelihood estimation. Unlike previous works on NNGP that targeted classification, here we apply the generalized NNGP to function approximation and to solving partial differential equations (PDEs). Specifically, we develop an analytical iteration formula to compute the covariance function of GP induced by deep NN with an error-function nonlinearity. We compare the performance of the generalized NNGP for function approximations and PDE solutions with those of GPs and fully-connected NNs. We observe that for smooth functions the generalized NNGP can yield the same order of accuracy with GP, while both NNGP and GP outperform deep NN. For non-smooth functions, the generalized NNGP is superior to GP and comparable or superior to deep NN.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Statistical and Nonlinear Physics, Model Architectures, Probabilistic & Bayesian Methods, Numerical Methods & Scientific Computing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Statistical and Nonlinear Physics

In Model Architectures

In Probabilistic & Bayesian Methods

In Numerical Methods & Scientific Computing

Citations in the corpus

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “PINNs have been extended to convolutional (Zhu et al., ∂x2 with initial condition u(x, 0) = f(x) and Dirichlet boundary conditions u(0, t) = 0 and u(L, t) = 0.”
fPINNs: Fractional Physics-Informed Neural Networks (2018, methodology): “In this paper, we focus on the NN approaches due to the high expressive power of NNs in function approximation [24, 25, 26, 27].”
B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data (2020, background): “We note that the prior for BNNs is a Gaussian process with zero mean as the width of each hidden layer goes to inﬁnity [16, 21, 22].”
Parametric Gaussian process regression for big data (2017, background): “process [19], may help capture the finer scales.”
Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems (2018, background): “For example, for the forward problem, some of the popular choices of machine learning tools are Gaussian process [3, 4, 5, 6, 7] and deep neural networks (DNNs) [8, 9, 10, 11, 12].”
Deep learning of vortex-induced vibrations (2018, background): “…of Gaussian process regression requires certain prior assumptions that may limit the representation capacity of the model and give rise to robustness/brittleness issues, especially for nonlinear problems [14], although this may be overcome by hybrid neural networks/Gaussian process algorithms [15].”
Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations (2018, methodology): “To this end, recent advances include using Gaussian process regression [17, 18, 19, 20, 21, 22] and deep neural networks (DNNs) [23, 24, 25, 12, 26] to solve forward problems as well as Bayesian estimation [27] and DNNs [28, 29] for inverse problems.”