Relu Deep Neural Networks and Linear Finite Elements

By Juncai He, Lin Li, Jinchao Xu, Chunyue Zheng. Journal of Computational Mathematics, 2018

364 citations recorded by the source corpus.

In this paper, we investigate the relationship between deep neural networks (DNN) with rectified linear unit (ReLU) function as the activation function and continuous piecewise linear (CPWL) functions, especially CPWL functions from the simplicial linear finite element method (FEM). We first consider the special case of FEM. By exploring the DNN representation of its nodal basis functions, we present a ReLU DNN representation of CPWL in FEM. We theoretically establish that at least $2$ hidden layers are needed in a ReLU DNN to represent any linear finite element functions in $\Omega \subseteq \mathbb{R}^d$ when $d\ge2$. Consequently, for $d=2,3$ which are often encountered in scientific and engineering computing, the minimal number of two hidden layers are necessary and sufficient for any CPWL function to be represented by a ReLU DNN. Then we include a detailed account on how a general CPWL in $\mathbb R^d$ can be represented by a ReLU DNN with at most $\lceil\log_2(d+1)\rceil$ hidden layers and we also give an estimation of the number of neurons in DNN that are needed in such a representation. Furthermore, using the relationship between DNN and FEM, we theoretically argue that a special class of DNN models with low bit-width are still expected to have an adequate representation power in applications. Finally, as a proof of concept, we present some numerical results for using ReLU DNNs to solve a two point boundary problem to demonstrate the potential of applying DNN for numerical solution of partial differential equations.

Subfields: Artificial Intelligence, Mechanics of Materials, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/51e75f774226f9b38646c6a7542eb543818723da

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Mechanics of Materials

In Statistical and Nonlinear Physics

Citations in the corpus

PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network (2018, background): “Neural networks have recently been proven effective in approximating multivariate functions in various scenarios [36, 37, 38, 39, 40].”
Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets (2019, methodology): “In particular, weight quantization of DNN have been extensively studied in the literature; see for examples (Li et al., 2016; Zhu et al., 2016; Li et al., 2017; Yin et al., 2016; 2018a; Hou & Kwok, 2018; He et al., 2018; Li & Hao, 2018).”
Collapse of Deep and Narrow Neural Nets (2018, background): “For example, in the work of [19], it was shown that deep and narrow ReLU networks are inevitable in constructing finite element’s basis functions.”
Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks (2019, background): “Although Xu et al. (2018) and Rahaman et al. (2018) showed the phenomena of the F-Principle on 1-d and 2-d synthetic datasets for the loss function of MSE, there is no empirical study demonstrating such phenomena work for high dimensional classiﬁcation problems on real datasets like MNIST/CIFAR10.”
Exact imposition of boundary conditions with distance functions in physics-informed deep neural networks (2021, methodology): “Since the PINN approximation that is composed by the ReLU activation function can exactly represent piecewise a ﬃ ne functions (Delaunay basis functions) [39], one can view the ReLU network solution as a variational r -adaptive ﬁnite element solution procedure.”
DeepXDE: A Deep Learning Library for Solving Differential Equations (2019, background): “Some researchers adopt the variational form of PDEs and minimize the corresponding energy functional [16, 20].”
KAN: Kolmogorov-Arnold Networks (2024, neutral): “To learn a function accurately, a model should not only learn the compositional structure ( external degrees of freedom), but should also approximate well the univariate functions ( internal degrees of freedom).”