Deep vs. shallow networks : An approximation theory perspective

By Hrushikesh Narhar Mhaskar, Tomaso A. Poggio. 2016

376 citations recorded by the source corpus.

The paper briefly reviews several recent results on hierarchical architectures for learning from examples, that may formally explain the conditions under which Deep Convolutional Neural Networks perform much better in function approximation problems than shallow, one-hidden layer architectures. The paper announces new results for a non-smooth activation function — the ReLU function — used in present-day neural networks, as well as for the Gaussian networks. We propose a new definition of relative dimension to encapsulate different notions of sparsity of a function class that can possibly be exploited by deep networks but not by shallow ones to drastically reduce the complexity required for approximation and learning.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Deep Neural Networks for Estimation and Inference: Application to Causal Effects and Other Semiparametric Estimands (2018, positive): “Using recent results (Mhaskar and Poggio (2016), Hanin (2019), Yarotsky (2018)), we can establish the following rate for very deep, ﬁxed-width MLPs.”
Learning Deep ResNet Blocks Sequentially using Boosting Theory (2017, background): “Theoretical analyses have further justified the power of deep networks (Mhaskar & Poggio, 2016) compared to shallow networks.”
Learning Sparse Neural Networks via Sensitivity-Driven Regularization (2018, positive): “However, once input-output relations are properly represented by a complex network, such a network may form a starting point in order to find a simpler, sparser, but sufficient architecture [4, 5].”
Nonparametric regression using deep neural networks with ReLU activation function (2017, background): “It is conceivable that for many of the problems for which neural networks perform well, a hidden hierarchical input-output relationship of the form (6) is present with small values t i ; cf. [35, 40].”
The loss surface and expressivity of deep convolutional neural networks (2017, neutral): “…to shallow networks in terms of expressiveness (Delalleau & Bengio, 2011; Telgarsky, 2016; 2015; Eldan & Shamir, 2016; Safran & Shamir, 2017; Yarotsky, 2016; Poggio et al., 2016; Liang & Srikant, 2017; Mhaskar & Poggio, 2016; Montufar et al., 2014; Pascanu et al., 2014; Raghu et al., 2017).”
Collapse of Deep and Narrow Neural Nets (2018, background): “…NN, but it cannot be approximated by any 2-layer network with the same accuracy irrespective of the activation function, unless its width is exponential in the dimension (see Mhaskar et al. (2017); Mhaskar & Poggio (2016); Delalleau & Bengio (2011); Poggio et al. (2017) for further discussions).”
Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review (2016, background): “Results in this direction are given in [30] , and more recently in [31] and [5] (see Theorem 3.1).”
The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks (2018, background): “Depth A large body of work has detailed the benefits of depth in neural networks (e.g. Montafur et al. (2014); Delalleau & Bengio (2011); Martens et al. (2013); Bianchini & Scarselli (2014); Shamir & Eldan (2015); Telgarsky (2015); Mhaskar & Shamir (2016)).”
Understanding deep learning (still) requires rethinking generalization (2021, background): “Much effort has gone into characterizing the expressivity of neural networks, for example, Cybenko 12, Mhaskar 25, Delalleau and Bengio 13, Mhaskar and Poggio 24, Eldan and Shamir 15, Telgarsky 40, Cohen and Shashua 11 Almost all of these results are at the “population level” showing what functions…”
Deep Autoencoder based Energy Method for the Bending, Vibration, and Buckling Analysis of Kirchhoff Plates (2020, background): “Recent studies show that DNNs render better approximations for nonlinear functions [37].”
On deep learning as a remedy for the curse of dimensionality in nonparametric regression (2019, background): “…focus in the theoretical analysis of approximation properties of neural networks is also on a possible theoretical advantage of multilayer feedforward neural networks in contrast to neural networks with only one hidden layer (cf., e.g., Eldan and Shamir (2015) and Mhaskar and Poggio (2016)).”
Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 2 Applications and Future Perspectives (2017, background): “However, the locality at each layer is a key factor which gives DCNNs an exponential advantage over shallow NNs in terms of the representation ability (Anselmi et al., 2015; Poggio et al., 2016; Mhaskar and Poggio, 2016).”
Impact of Fully Connected Layers on Performance of Convolutional Neural Networks for Image Classification (2019, methodology): “The eﬀect of deep or shallow networks on diﬀerent kind of datasets is well explored in the literature to study the behavioral interrelationship between depth of dataset and the CNN [12, 13].”
The Kolmogorov-Arnold representation theorem revisited (2020, background): “Approximation properties of deep neural networks for composed functions are studied in [12, 25, 21, 13, 26, 3, 23, 27, 6].”
Understanding deep learning requires rethinking generalization (2016, background): “Much effort has gone into characterizing the expressivity of neural networks, e.g, Cybenko (1989); Mhaskar (1993); Delalleau & Bengio (2011); Mhaskar & Poggio (2016); Eldan & Shamir (2016); Telgarsky (2016); Cohen & Shashua (2016).”