On the Ability of Neural Nets to Express Distributions

By Holden Lee, Rong Ge, Tengyu Ma, Andrej Risteski, Sanjeev Arora. 2017

89 citations recorded by the source corpus.

Deep neural nets have caused a revolution in many classification tasks. A related ongoing revolution---also theoretically not understood---concerns their ability to serve as generative models for complicated types of data such as images and texts. These models are trained using ideas like variational autoencoders and Generative Adversarial Networks. We take a first cut at explaining the expressivity of multilayer nets by giving a sufficient criterion for a function to be approximable by a neural network with $n$ hidden layers. A key ingredient is Barron's Theorem \cite{Barron1993}, which gives a Fourier criterion for approximability of a function by a neural network with 1 hidden layer. We show that a composition of $n$ functions which satisfy certain Fourier conditions ("Barron functions") can be approximated by a $n+1$-layer neural network. For probability distributions, this translates into a criterion for a probability distribution to be approximable in Wasserstein distance---a natural metric on probability distributions---by a neural network applied to a fixed base distribution (e.g., multivariate gaussian). Building up recent lower bound work, we also give an example function that shows that composition of Barron functions is more expressive than Barron functions alone.

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “their ability to build complex functions with a much smaller number of parameters than needed for shallow networks [9–13].”
On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization (2018, background): “Superior expressiveness of deeper networks, long suspected, is now conﬁrmed by theory, albeit for fairly limited learning problems (Eldan & Shamir, 2015; Raghu et al., 2016; Lee et al., 2017; Cohen et al., 2017; Daniely, 2017; Arora et al., 2018).”
Theoretical Guarantees for Model Auditing with Finite Adversaries (2019, background): “It is important to remark that our bounds depend heavily on the Barron constant, a quantity used to quantify the approximating power of neural networks [16,17].”
Emergence of Invariance and Disentanglement in Deep Representations (2017, background): “Other theoretical eﬀorts focus on complexity considerations, and explain the success of deep networks by ways of statistical or computational eﬃciency (Lee et al., 2017; Bengio et al., 2009; LeCun, 2012).”
Greedy Layerwise Learning Can Scale to ImageNet (2018, background): “ρWθ̂1X0 is obtained from an initial bounded distribution X0, Cfj is the Barron Constant (as described in (Lee et al., 2017)) of fj , δρ a constant depending on ρ and an estimation error.”
Emergence of invariance and disentangling in deep representations (2017, background): “Other theoretical efforts focus on complexity considerations, and explain the success of deep networks by ways of statistical or computational efficiency (Lee et al., 2017; Bengio et al., 2009; LeCun, 2012).”