ResNet with one-neuron hidden layers is a Universal Approximator

By Hongzhou Lin, Stefanie Sabrina Jegelka. 2018

257 citations recorded by the source corpus.

We demonstrate that a very deep ResNet with stacked modules with one neuron per hidden layer and ReLU activation functions can uniformly approximate any Lebesgue integrable function in $d$ dimensions, i.e. $\ell_1(\mathbb{R}^d)$. Because of the identity mapping inherent to ResNets, our network has alternating layers of dimension one and $d$. This stands in sharp contrast to fully connected networks, which are not universal approximators if their width is the input dimension $d$ [Lu et al, 2017; Hanin and Sellke, 2017]. Hence, our result implies an increase in representational power for narrow deep networks by the ResNet architecture.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Augmented Neural ODEs (2019, methodology): “To deﬁne a model from R d to R , we follow the example given in Lin & Jegelka (2018) for ResNets.”
Choose a Transformer: Fourier or Galerkin (2021, background): “…approximator such as a deep neural network to approximate functions/operators, following the discussion in Section D.2, the “consistency” of an approximation class embodies itself as the Universal Approximation Theorem (UAT, e.g., see [42, 18], and [58, 67, 77, 99] for modern expositions).”
The Expressive Power of a Class of Normalizing Flow Models (2020, background): “According to (Lin and Jegelka, 2018), piecewise constant functions supported on a finite union of compact intervals can approximate any Lebesgue-integrable function, so do densities supported on a finite union of intervals.”
A Generalization Theory of Gradient Descent for Learning Over-parameterized Deep ReLU Networks (2019, background): “Lin and Jegelka (2018) proved that ResNet (He et al., 2016) with one hidden node per layer is a universal approximator to any Lebesgue integrable function.”
Universal Differential Equations for Scientific Machine Learning (2020, background): “[20] Hongzhou Lin and Stefanie Jegelka.”
Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth (2020, background): “Neural network models of different depth and width have been studied through the lens of universal approximation theorems (Cybenko, 1989; Hornik, 1991; Pinkus, 1999; Lu et al., 2017; Hanin & Sellke, 2017; Lin & Jegelka, 2018) and functional expressivity (Telgarsky, 2015; Raghu et al., 2017b).”
BayesFlow: Learning Complex Stochastic Models With Invertible Neural Networks (2020, background): “The benefits of neural network depth has been confirmed both in theory [31] and practice [17], so we expect better performance in complex settings with increasing number of ACBs.”
Constrained Reinforcement Learning Has Zero Duality Gap (2019, background): “If we consider, for instance, a neural network—which are universal function approximators [12, 25–28]—the loss in optimality can be made arbitrarily small.”
On Measuring Excess Capacity in Neural Networks (2022, background): “For example, while residual networks have been studied extensively, theory mostly focuses on expressivity or optimization aspects [3, 18, 28, 44].”
Are Transformers universal approximators of sequence-to-sequence functions? (2019, methodology): “Other results focusing on depth appeared more recently (Lu et al., 2017; Hanin & Sellke, 2017; Lin & Jegelka, 2018).”
Universal Approximation with Deep Narrow Networks (2019, result): “tion function are dense in C(K;Rm)for K⊆ Rn compact, and require only width n+m. Lu et al. (2017) have shownthat deep narrow networks withthe ReLUactivation function aredense inL1(Rn),with width n+4. Lin and Jegelka (2018) have shown that a particular description of residual networks, with the ReLU ”
Gradient descent optimizes over-parameterized deep ReLU networks (2019, background): “Moreover, it is also interestingtoexplorehowouroptimizationguaranteesofover-parameterizedneuralnetworks canbeintegratedwithexistinguniversalapproximationabilityresultssuchasHornik(1991), Telgarsky (2016), Lin and Jegelka (2018), Zhou (2019).”
On Interpretability of Artificial Neural Networks: A Survey (2020, background): “[178] showed that residual connections can render a neural network to manifest an ensemble-like behavior.”
Deep learning based brain tumor segmentation: a survey (2020, background): “Neural networks are able to learn high dimensional hierarchical features and approximate any continuous functions [24], [25].”