Universal Approximation with Deep Narrow Networks
By Patrick Kidger, Terry Lyons. Annual Conference Computational Learning Theory, 2019
430 citations recorded by the source corpus.
The classical Universal Approximation Theorem holds for neural networks of arbitrary width and bounded depth. Here we consider the natural `dual' scenario for networks of bounded width and arbitrary depth. Precisely, let $n$ be the number of inputs neurons, $m$ be the number of output neurons, and let $\rho$ be any nonaffine continuous function, with a continuous nonzero derivative at some point. Then we show that the class of neural networks of arbitrary depth, width $n + m + 2$, and activation function $\rho$, is dense in $C(K; \mathbb{R}^m)$ for $K \subseteq \mathbb{R}^n$ with $K$ compact. This covers every activation function possible to use in practice, and also includes polynomial activation functions, which is unlike the classical version of the theorem, and provides a qualitative difference between deep narrow networks and shallow wide networks. We then consider several extensions of this result. In particular we consider nowhere differentiable activation functions, density in noncompact domains with respect to the $L^p$-norm, and how the width may be reduced to just $n + m + 1$ for `most' activation functions.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/061f90dff7b2b42e6ba75ec3fd86e9ce7fd18e8d
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Patrick Kidger
- Samy Wu Fung
- Chi Chen
- Alok N. Choudhary
- Terry Lyons
- Daniel Mckenzie
- Stanley J. Osher
- Howard Heaton
Citations in the corpus
- Recent advances and applications of deep learning methods in materials science (2021, background): “According to the “ universal approximation theorem ” 50,51 , neural networks can approximate any function to arbitrary accuracy.”
- Neural Controlled Differential Equations for Irregular Time Series (2020, background): “Classical universal approximation results for neural networks [68, 69] then yield that By the definition of the natural cubic spline topology on”
- JFB: Jacobian-Free Backpropagation for Implicit Networks (2021, background): “Universal approximation properties of implicit networks then follow immediately from such properties of conventional deep neural models ( e.g. see (Cs´aji et al. 2001; Lu et al. 2017; Kidger and Lyons 2020)).”