Approximation rates for neural networks with general activation functions
By Jonathan W. Siegel, Jinchao Xu. 2019
191 citations recorded by the source corpus.
We prove some new results concerning the approximation rate of neural networks with general activation functions. Our first result concerns the rate of approximation of a two layer neural network with a polynomially-decaying non-sigmoidal activation function. We extend the dimension independent approximation rates previously obtained to this new class of activation functions. Our second result gives a weaker, but still dimension independent, approximation rate for a larger class of activation functions, removing the polynomial decay assumption. This result applies to any bounded, integrable activation function. Finally, we show that a stratified sampling approach can be used to improve the approximation rate for polynomially decaying activation functions under mild additional assumptions.
Subfields: Artificial Intelligence, Control and Systems Engineering, Statistical and Nonlinear Physics, Model Architectures, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Control and Systems Engineering
In Statistical and Nonlinear Physics
In Model Architectures
In Scaling & Learning Theory
Citations in the corpus
- Training Compute-Optimal Large Language Models (2022, neutral): “On the set of two-layer neural networks , it is expected to be proportional to 1 𝑁 1 / 2 (Siegel and Xu, 2020).”
- Choose a Transformer: Fourier or Galerkin (2021, background): “…approximator such as a deep neural network to approximate functions/operators, following the discussion in Section D.2, the “consistency” of an approximation class embodies itself as the Universal Approximation Theorem (UAT, e.g., see [42, 18], and [58, 67, 77, 99] for modern expositions).”