Approximation by superpositions of a sigmoidal function

By George V. Cybenko. 1989

In this paper we demonstrate that finite linear combinations of compositions of a fixed, univariate function and a set of affine functionals can uniformly approximate any continuous function ofn real variables with support in the unit hypercube; only mild conditions are imposed on the univariate function. Our results settle an open question about representability in the class of single hidden layer neural networks. In particular, we show that arbitrary decision regions can be arbitrarily well approximated by continuous feedforward neural networks with only a single internal, hidden layer and any continuous sigmoidal nonlinearity. The paper discusses approximation properties of other possible types of nonlinearities that might be implemented by artificial neural networks.

Subfields: Model Architectures, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Statistical Learning Theory

Citations in the corpus

On the Number of Linear Regions of Deep Neural Networks (2014, background): “A well-known result states that a feedforward neural network with a single, huge, hidden layer is a universal approximator of Borel measurable functions (see Hornik et al. 1989, Cybenko 1989).”
Scattering Networks for Hybrid Representation Learning (2018, background): “Many theoretical arguments of deep learning rely on the universal approximation theorem [14].”
A Closer Look at Memorization in Deep Networks (2017, background): “On the contrary, deep nets are known to be universal ap-proximators, capable of representing arbitrarily complex functions given sufﬁcient capacity (Cybenko, 1989; Hornik et al., 1989).”
Do Deep Nets Really Need to be Deep? (2013, background): “On the TIMIT phoneme recognition and CIFAR-10 image recognition tasks, shallow nets can be trained that perform similarly to complex, well-engineered, deeper convolutional architectures.”
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks (2015, background): “The existence of function F ∗ is guaranteed by the universality theorem for neural networks [33], which states that with enough neurons and enough training points, one can approximate any continuous function with arbitrary precision.”
Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019 (2019, background): “Most commonly used nonlinear activation functions are: sigmoid (Equation 2) [40], hyperbolic tangent (Equation 3) [41], Rectified Linear Unit (ReLU) (Equation 4) [42], leaky-ReLU (Equation 5) [43], swish (Equation 6) [44], and softmax (Equation 7) [39].”
Theoretical Guarantees for Model Auditing with Finite Adversaries (2019, background): “The latter results characterize how well certain functions can be approximated using neural networks, see [16, 17, 20–25] and references therein.”
Wireless Network Intelligence at the Edge (2018, background): “The proof is provided for shallow NNs [83], [84] and recently for deep NNs [85].”
Robust Large Margin Deep Neural Networks (2016, background): “For example, the works in [8], [9] showed that neural networks with a single hidden layer—shallow networks—can approximate any measurable Borel function.”
Learning explanations that are hard to vary (2020, background): “One well known version of the universal approximation function theorem [Cybenko, 1989] shows that a one-hidden-layer neural network with sigmoid activations and enough neurons can approximate any function f∗ : [0, 1]→ R.”
Generalization in anti-causal learning (2018, background): “…for example within the empirical risk minimization (ERM) framework (Vapnik, 1998), paired with universal function approximation guarantees for neural nets (Hornik et al., 1989; Cybenko, 1989) prompt the conventional wisdom that “with enough data and powerful models anything can be learned ”.”
An overview of deep learning in medical imaging focusing on MRI (2018, methodology): “According to the famous universal approximation theorem for artificial neural networks [46, 47, 48, 49], ANNs are mathematically able to approximate any continuous function defined on compact subspaces of R, using finitely many neurons.”
Causal Generative Neural Networks (2017, methodology): “Following the universal approximation theorem [Cybenko, 1989], as f i is a continuous function over a compact of , there exists a neural net ˆ f i such that (cid:107) − (cid:107) Thus Eq.”
Physics informed deep learning for computational elastodynamics without labeled data (2020, methodology): “This convergence property of PINN is guaranteed by the universal approximation theorem [49].”