Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review

By Tomaso A. Poggio, Hrushikesh Narhar Mhaskar, Lorenzo Rosasco, Brando Miranda, Qianli Liao. 2016

657 citations recorded by the source corpus.

The paper reviews and extends an emerging body of theoretical results on deep learning including the conditions under which it can be exponentially better than shallow learning. A class of deep convolutional networks represent an important special case of these conditions, though weight sharing is not the main reason for their exponential advantage. Implications of a few key theorems are discussed, together with new results, open problems and conjectures.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Citations in the corpus

Bayesian Nonparametric Federated Learning of Neural Networks (2019, neutral): “The matched neurons are then aggregated via Proposition 1 to form the global model. networks with moderate layer widths are known to be bene-ﬁcial both practically (LeCun et al., 2015) and theoretically (Poggio et al., 2017).”
Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting (2019, neutral): “f(x) =  A1 sin(πx/6) + 72 +Nx x ∈ [0, 12), A2 sin(πx/6) + 72 +Nx x ∈ [12, 24), A3 sin(πx/6) + 72 +Nx x ∈ [24, t0), A4 sin(πx/12) + 72 +Nx x ∈ [t0, t0 + 24).”
Deep Learning for Predicting Asset Returns (2018, background): “Deep networks, as opposed to shallow ones, can achieve out-of-sample performance gains versus linear additive models, while avoiding the curse of dimensionality, for example, see Poggio et al. (2017).”
Generalization in Deep Learning (2017, background): “From the viewpoint of compact representation related to small capacity, it has been shown that deep model classes have an exponential advantage to represent certain natural target functions when compared to shallow model classes (Pascanu et al., 2014; Montufar et al., 2014; Livni et al., 2014; Telga”
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes (2018, background): “…properties of NNs which are believed to make them well suited to modeling the world include: hierarchy and compositionality (Lin et al., 2017; Poggio et al., 2017), Markovian dynamics (Tiňo et al., 2004; 2007), and equivariances in time and space for RNNs (Werbos, 1988) and CNNs (Fukushima…”
Nonparametric regression using deep neural networks with ReLU activation function (2017, background): “It is conceivable that for many of the problems for which neural networks perform well, a hidden hierarchical input-output relationship of the form (6) is present with small values t i ; cf. [35, 40].”
Deep learning generalizes because the parameter-function map is biased towards simple functions (2018, background): “Many empirical schemes such as dropout (Srivastava et al. (2014)), weight decay [cite] early stopping (Morgan & Bourlard (1990)), have been proposed as sources of regularization that effectively lower C(H). However, in an important recent paper (Zhang et al. (2017a)), it was explicitly demonstrated ”
Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 2 Applications and Future Perspectives (2017, background): “…within the theory of TNs, which often employ physical constraints of locality, symmetry, compositional hierarchical functions, entropy, and polynomial log-probability, that are imposed on the measurements or input training data [Chen et al., 2017, Lin and Tegmark, 2016, Poggio et al., 2016].”
Machine learning \& artificial intelligence in the quantum domain (2017, background): “First, they may reduce the number of parameters (Poggio et al., 2017).”
Depth Creates No Bad Local Minima (2017, background): “More accurately, a key aspect of the model classes in deep learning is the generalization property; despite its great expressivity, deep learning model classes can maintain great generalization properties (Livni et al., 2014; Mhaskar et al., 2016; Poggio et al., 2016).”
PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network (2018, background): “Neural networks have recently been proven effective in approximating multivariate functions in various scenarios [36, 37, 38, 39, 40].”
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning (2018, background): “different Heavy-Tailed Universality classes [46, 32, 18, 25, 20, 5, 111, 7, 38, 17, 90, 8, 99]; other work on RMT approaches [133, 120, 114, 112, 87, 87, 137, 85, 126]; other work on statistical physics approaches [132, 52, 117, 125, 137, 119, 113]; work on fitting to noisy versus reliable signal [1”
When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization? (2021, background): “The term “tree like function space” is taken from one of the most related works [18], and a famous paper on neural network approximation [20].”