Learning Real and Boolean Functions: When Is Deep Better Than Shallow

By Hrushikesh Narhar Mhaskar, Qianli Liao, Tomaso A. Poggio. 2016

61 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep Learning without Poor Local Minima (2016, background): “In addition to its practical success, theoretical results have shown that deep learning is attractive in terms of its generalization properties (Livni et al. , 2014; Mhaskar et al. , 2016).”
On the Expressive Power of Deep Neural Networks (2016, methodology): “Armed with this notion of trajectories, we can begin to define measures of expressivity of a network FA(x;W ) over trajectories x(t).”
Exponential expressivity in deep neural networks through transient chaos (2016, background): “For example, [9] focused on ReLU nonlinearities and number of linear regions as a complexity measure, while [10] focused on sum-product networks and the number of monomials as complexity measure, and [14] focused on Pfaffian nonlinearities and topological measures of complexity, like the sum of Bett”
Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex (2016, background): “This may oﬀer a new perspective on the theoretical pursuit of the long-standing question “why is deep better than shallow” [22, 21].”
Inductive Bias of Deep Convolutional Networks through Pooling Geometry (2016, background): “In recent years, a large body of research was devoted to proving existence of depth efﬁciency under different types of architectures (see for example [9, 23, 21, 28, 10, 24, 20]).”
Error bounds for approximations with deep ReLU networks (2016, methodology): “Much less seems to be known about deep networks in this setting, though Mhaskar et al. 2016, 2016 have recently introduced functional spaces constructed using deep dependency graphs and obtained expressiveness bounds for related deep networks.”
Optimal approximation of continuous functions by very deep ReLU networks (2018, background): “…Scarselli, 2014), Vapnik-Chervonenkis dimension (Bartlett et al., 1998; Sakurai, 1999) and fat-shattering dimension (Kearns and Schapire, 1990; Anthony and Bartlett, 2009), hierarchical decompositions of functions (Mhaskar et al., 2016), information theory (Petersen and Voigtlaender, 2017), etc.”
Bounding and Counting Linear Regions of Deep Neural Networks (2017, background): “Finally, Arora et al. (2018) improve the lower bound by providing a family of ReLU DNNs with an exponential number of regions for ﬁxed size and depth.”
Deep vs. shallow networks : An approximation theory perspective (2016, background): “In this paper we review and extend a theoretical framework that we have introduced very recently to address the ﬁrst question [20].”
Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review (2016, background): “This paper describes and extends results pre-1 sented in [21, 22, 23] and in [24, 4] which derive new upper bounds for the approximation by deep networks of certain important classes of functions which avoid the curse of dimensionality.”
Depth Creates No Bad Local Minima (2017, background): “More accurately, a key aspect of the model classes in deep learning is the generalization property; despite its great expressivity, deep learning model classes can maintain great generalization properties (Livni et al., 2014; Mhaskar et al., 2016; Poggio et al., 2016).”