On the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures

By Monica Bianchini, Franco Scarselli. 2014

526 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Scaling & Learning Theory

Citations in the corpus

Training Very Deep Networks (2015, background): “Related observations hold for Boolean circuits [8, 9] and modern neural networks [10, 11, 12].”
The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “their ability to build complex functions with a much smaller number of parameters than needed for shallow networks [9–13].”
Recurrent Highway Networks (2016, background): “Theoretical evidence indicates that deeper networks can be exponentially more efﬁcient at representing certain function classes (see e.g. [2] and references therein).”
Resnet in Resnet: Generalizing Residual Architectures (2016, background): “The ﬁxed size layer structure of the residual block modules also enforces that residuals must be learned by shallow subnetworks, despite evidence that deeper networks are more expressive (Bianchini & Scarselli, 2014).”
Human-grounded Evaluations of Explanation Methods for Text Classification (2019, background): “Further studies of this quality of the DTs would be useful for some applications, e.g., measuring model complexity (Bianchini and Scarselli, 2014) and model compression (Cheng et al., 2018).”
Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality (2018, background): “It is shown that the expressive power of deep neural network grows exponentially against the number of layers (Montufar et al., 2014; Bianchini & Scarselli, 2014; Cohen et al., 2016; Cohen & Shashua, 2016; Poole et al., 2016) where the “expressive power” is deﬁned by several ways.”
Efficient Representation of Low-Dimensional Manifolds using Deep Networks (2016, background): “However, recent work demonstrates that deep networks can be exponentially more efﬁcient in representing certain functions (Bianchini & Scarselli, 2014; Telgarsky, 2015; Eldan & Shamir, 2015; Delalleau & Bengio, 2011; Montufar et al., 2014; Cohen et al., 2015).”
On simplicity and complexity in the brave new world of large-scale neuroscience. (2015, background): “The theory of deep learning is still in its infancy, but some examples of general statements one can make about deep circuits without actually simulating them include how their functional complexity scales with depth [70, 71], how their synaptic weights, over time, acquire statistical structures in ”
On the Depth of Deep Neural Networks: A Theoretical View (2015, methodology): “In (Bianchini and Scarselli 2014; Montufar et al. 2014), it was demonstrated that deeper nets could represent more complex functions than shallower nets in terms of maximal number of linear regions and Betti numbers.”
The Effect of Network Width on the Performance of Large-batch Training (2018, background): “[25] suggests that deep networks have larger complexity than wide networks and thus may be able to obtain better models.”
On the complexity of shallow and deep neural network classifiers (2014, background): “Section 2 presents the main theoretical results, whose proofs, omitted here for space limitations, can be found in [7].”
NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results (2017, background): “The Haar wavelet transform [4] can annihilate the smoothly varying signals while maintaining the image edges, which results in simpler manifolds.”
Self-Attention ConvLSTM for Spatiotemporal Prediction (2020, background): “All of them stack convolution layers to obtain spatial dependences since deeper networks can be exponentially more efﬁcient in capturing both spatial and temporal dependences (Bianchini and Scarselli 2014; Pascanu, Mikolov, and Bengio 2013).”