On the Expressive Power of Deep Architectures

By Yoshua Bengio, Olivier Delalleau. 2011

312 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Scaling & Learning Theory

Citations in the corpus

Better Mixing via Deep Representations (2012, background): “specific function families have shown that choosing a sufficient depth of representation can yield exponential benefits, in terms of size of the model, to represent some functions (H̊astad, 1986; H̊astad and Goldmann, 1991; Bengio et al., 2006; Bengio and LeCun, 2007; Bengio and Delalleau, 2011).”
Representation Learning: A Review and New Perspectives (2012, background): “Unfortunately, the number of different regions represented (equal to the number of leaves of the tree) still only grows linearly with the number of parameters used to specify it (Bengio and Delalleau, 2011).”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “Theoretical arguments support a potentially exponential gain in expressive power of deeper architectures (Bengio, 2009; Bengio and Delalleau, 2011) and it would be plausible that with this added expressive power coming from the combinatorics of composed reuse of sub-26 functions could come a…”
Efficient Representations for Lifelong Learning and Autoencoding (2014, background): “These commonalities could be a single low-dimensional or sparse representation, a collection of multiple low-dimensional or sparse representations, or some combination or hierarchy, such as in Deep Learning [9, 10].”
Swapout: Learning an ensemble of deep architectures (2016, background): “Bengio and Dellaleau argue that circuit efﬁciency constraints suggest increasing depth is important, because there are functions that require exponentially large shallow networks to compute [1].”
The Open World of Micro-Videos (2016, background): “With the advent of deep architectures that appear capable of transferring knowledge across imbalanced classes [30], we think the time is right to (re)consider learning in the open-world!”
Learning Deep Neural Network Representations for Koopman Operators of Nonlinear Dynamical Systems (2017, background): “This may be because the deeper the neural network, the more efficiently it encodes complex nonlinear functions [19], [18].”
On the complexity of shallow and deep neural network classifiers (2014, background): “The existing results are limited to neural networks with boolean inputs, implementing logical gates or threshold activations, and to sum–product networks [5, 6].”
Model complexity of deep learning: a survey (2021, methodology): “In order to represent a large amount of knowledge and obtain high accuracy, a model with a high expressive capacity, a large degree of freedom and a large training set is required [13].”
PhosContext2vec: a distributed representation of residue-level sequence contexts and its application to general and kinase-specific phosphorylation site prediction (2018, result): “The SVM algorithm used has a comparatively less expressive power 39 , which may partially explain the inferior performance of PhosContext2vec compared to that of MusiteDeep.”
Deep Learning on FPGAs: Past, Present, and Future (2016, background): “Conventional approaches to artiﬁcial intelligence focused on using computation to solve problems analytically, requiring explicit knowledge about a given domain [10].”
On the Expressive Efficiency of Sum Product Networks (2014, background): “h each l ayer of depth added. This “depth hierarchy” property has never before been shown to hold for another non-trivial deep model, despite the widespread belief that it does for most of them [e.g. Bengio and Delalleau, 2011]. In addition to these two major results, we also make numerous other con”
Dual autoencoders features for imbalance classification problem (2016, background): “However, a single layer autoencoder may not be powerful enough to learn useful features [3,19,32] while stacked autoencoders with more than two layers may not yield much better representation at the cost of slower training time.”
On the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures (2014, unknown): “The existing results are limited to neural networks with boolean inputs and units, implementing logical gates or threshold activation functions, and to sum-product networks [22], [23].”
A Deep Learning Approach to Data-driven Parameterizations for Statistical Parametric Speech Synthesis (2014, result): “This is in line with deep learning theory[29].”
Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives (2012, background): “Unfortunately, the number of different regions represented (equal to the number of leaves of the tree) still only grows linearly with the number of parameters used to specify it (Bengio and Delalleau, 2011).”
Wind Turbine Gearbox Failure Identification With Deep Neural Networks (2017, background): “The appropriate configuration of the DNN allows it to approximate complicated functions with a linear number of hidden units, while NNs might require an exponential number of hidden units to achieve the similar performance [26].”
Deep learning for healthcare applications based on physiological signals: A review (2018, background): “This change percolates through to all application areas of machine learning, such as computer vision, speech recognition, natural language processing and indeed diagnosis support [36].”