Shallow vs. Deep Sum-Product Networks

By Olivier Delalleau, Yoshua Bengio. 2011

349 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

On the Number of Linear Regions of Deep Neural Networks (2014, background): “Recently, Delalleau and Bengio (2011) showed that a shallow network requires exponentially many more sum-product hidden units 1 than a deep sum-product network in order to compute certain families of polynomials.”
Understanding deep learning requires rethinking generalization (2016, background): “In contrast to existing depth separations (Delalleau & Bengio, 2011; Eldan & Shamir, 2016; Telgarsky, 2016; Cohen & Shashua, 2016) in function space, our result shows that even depth-2 networks of linear size can already represent any labeling of the training data.”
How to Construct Deep Recurrent Neural Networks (2013, background): “For instance, it has been shown by Delalleau and Bengio (2011) that a deep sum-product network may require exponentially less units to represent the same function compared to a shallow sum-product network.”
Exponential expressivity in deep neural networks through transient chaos (2016, background): “For example, [9] focused on ReLU nonlinearities and number of linear regions as a complexity measure, while [10] focused on sum-product networks and the number of monomials as complexity measure, and [14] focused on Pfaffian nonlinearities and topological measures of complexity, like the sum of Bett”
The Power of Depth for Feedforward Neural Networks (2015, background): “Delalleau and Bengio [2011], Martens and Medabalimi [2014] and Cohen et al. [2015] show strong approximation hardness results for certain neural network architectures (such as polynomials or representing a certain tensor structure), which are however fundamentally different than the standard neural…”
Benefits of Depth in Neural Networks (2016, methodology): “For architectures closer to neural networks, sum-product networks (summation and product nodes) have been analyzed by Bengio and Delalleau (2011) and more recently Martens and Medabalimi (2015), and networks of linear threshold functions in 2 and 3 layers by Kane and Williams (2015); note that both ”
Demystifying Parallel and Distributed Deep Learning (2018, background): “ after another, we create deep networks (as shown in Fig.7b) that can approximatearbitrarilycomplex continuousfunctions. Whiletheexact classofexpressiblefunctions is currently an open problem, results[47,59] show that neural network depth can reduce breadth requirements exponentially with each addit”
Hessian-based Analysis of Large Batch Training and Robustness to Adversaries (2018, background): “The diversity of the different problems that a DNN can be used for, has been related to their efficiency in function approximation [25, 7, 21, 1].”
Inductive Bias of Deep Convolutional Networks through Pooling Geometry (2016, background): “In recent years, a large body of research was devoted to proving existence of depth efﬁciency under different types of architectures (see for example [9, 23, 21, 28, 10, 24, 20]).”
Toward an Integration of Deep Learning and Neuroscience (2016, background): “…persistent states (Hochreiter and Schmidhuber, 1997; Chung et al., 2014), more complex elementary units such as “capsules” and other structures (Delalleau and Bengio, 2011; Hinton et al., 2011; Tang et al., 2012; Livni et al., 2013 ), content addressable (Graves et al., 2014; Weston et al.,…”
Are All Layers Created Equal? (2019, background): “…examines whether deep networks can have superior representation power than shallow ones with the same number of units or edges (Pinkus, 1999; Delalleau and Bengio, 2011; Montufar et al., 2014; Telgarsky, 2016; Shaham et al., 2015; Eldan and Shamir, 2015; Mhaskar and Poggio, 2016; Rolnick…”
Model complexity of deep learning: a survey (2021, background): “R ao no rm [7 4] • • A ny de ep m od el s M ax #s am pl es to ze ro tr ai ni ng er ro r H ig hca pa ci ty lo w -r ea lit y ph e- no m en on [3 7] • • FC N N w ith σ (z ) = {R eL U } #l in ea r re gi on s, vo lu m e of re gi on bo un da ri es study model complexity of fully connected feedforward neur”
A survey of the recent architectures of deep convolutional neural networks (2019, background): “In this regard, Bengio and Delalleau (Delalleau and Bengio 2011) suggested that deeper networks can maintain the expressive power of the network at a reduced cost (Wang and Raj 2017).”
Error bounds for approximations with deep ReLU networks (2016, background): “Delalleau and Bengio 2011 consider sum-product networks and prove for certain classes of polynomials that they are much more easily represented by deep networks than by shallow networks.”
Efficient Representation of Low-Dimensional Manifolds using Deep Networks (2016, background): “However, recent work demonstrates that deep networks can be exponentially more efﬁcient in representing certain functions (Bianchini & Scarselli, 2014; Telgarsky, 2015; Eldan & Shamir, 2015; Delalleau & Bengio, 2011; Montufar et al., 2014; Cohen et al., 2015).”
On the Origin of Deep Learning (2017, background): “Moving from circuits to neural networks, Delalleau and Bengio (2011) compared deep and shallow sum-product neural networks.”
Why Deep Neural Networks for Function Approximation? (2016, background): “Delalleau & Bengio (2011) have shown that there exist deep sum-product networks which cannot be approximated by shallow sum-product networks unless they use an exponentially larger amount of units or neurons.”