Deep Network Approximation for Smooth Functions

By Jianfeng Lu, Zuowei Shen, Haizhao Yang, Shijun Zhang. SIAM Journal on Mathematical Analysis, 2020

314 citations recorded by the source corpus.

This paper establishes optimal approximation error characterization of deep ReLU networks for smooth functions in terms of both width and depth simultaneously. To that end, we first prove that multivariate polynomials can be approximated by deep ReLU networks of width $\mathcal{O}(N)$ and depth $\mathcal{O}(L)$ with an approximation error $\mathcal{O}(N^{-L})$. Through local Taylor expansions and their deep ReLU network approximations, we show that deep ReLU networks of width $\mathcal{O}(N\ln N)$ and depth $\mathcal{O}(L\ln L)$ can approximate $f\in C^s([0,1]^d)$ with a nearly optimal approximation rate $\mathcal{O}(\|f\|_{C^s([0,1]^d)}N^{-2s/d}L^{-2s/d})$. Our estimate is non-asymptotic in the sense that it is valid for arbitrary width and depth specified by $N\in\mathbb{N}^+$ and $L\in\mathbb{N}^+$, respectively.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Numerical Analysis

View Paper: https://www.semanticscholar.org/paper/0046c91cc46482817634d31f299072b87cf7167a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

  1. Ronald A. DeVore
  2. Guergana Petrova
  3. W. E
  4. Boris Hanin
  5. Alexander Kurganov
  6. Albert Cohen
  7. Rob P. Stevenson
  8. Jesús María Sanz-Serna

In Computer Graphics and Computer-Aided Design

  1. Ronald A. DeVore

In Numerical Analysis

  1. Guergana Petrova
  2. W. E
  3. Ronald A. DeVore
  4. Jesús María Sanz-Serna
  5. Eric Vanden-Eijnden
  6. Christoph Schwab
  7. E. Weinan

Citations in the corpus