Understanding Deep Architectures using a Recursive Convolutional Network

By David Eigen, Jason Tyler Rolfe, Rob Fergus, Yann LeCun. 2013

148 citations recorded by the source corpus.

A key challenge in designing convolutional network models is sizing them appropriately. Many factors are involved in these decisions, including number of layers, feature maps, kernel sizes, etc. Complicating this further is the fact that each of these influence not only the numbers and dimensions of the activation units, but also the total number of parameters. In this paper we focus on assessing the independent contributions of three of these linked variables: The numbers of layers, feature maps, and parameters. To accomplish this, we employ a recursive convolutional network whose weights are tied between layers; this allows us to vary each of the three factors in a controlled setting. We find that while increasing the numbers of layers and parameters each have clear benefit, the number of feature maps (and hence dimensionality of the representation) appears ancillary, and finds most of its benefit through the introduction of more weights. Our results (i) empirically confirm the notion that adding layers alone increases computational power, within the context of convolutional layers, and (ii) suggest that precise sizing of convolutional feature map dimensions is itself of little concern; more attention should be paid to the number of parameters in these layers instead.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015, background): “e large and overly deep models will ultimately degrade). Because of the possible degradation, we choose not to further increase the depth of these large models. On the other hand, the recent research [5] on small datasets suggests that the accuracy should improve from the increased number of paramet”
Convolutional neural networks at constrained time cost (2014, background): “In [5], the accuracy of tied/untied CNNs is evaluated with various width, depth, and numbers of parameters.”
Deeply-Supervised Nets (2014, background): “These open questions arise in conjunction with various aspects of current DL frameworks: the features learned at hidden layers (and at early hidden layers in particular) are not always “transparent” in their meaning and at times display reduced discrimination ability [29]; the presence of “exploding”
Aggregated Residual Transformations for Deep Neural Networks (2016, background): “This is not only interesting in practice, but more importantly, the complexity and number of parameters represent inherent capacity of models and thus are often investigated as fundamental properties of deep networks [8].”
Do Deep Nets Really Need to be Deep? (2013, methodology): “Similarly, [7] used a base convolution and pooling layer to study different deep architectures.”
Learning Deep Structured Models (2014, background): “Similarly state-of-the-art results have been demonstrated for natural language processing, e.g., in [26, 5, 6].”
Convergent Learning: Do different neural networks learn the same representations? (2015, background): “…2013; Montavon et al., 2011; Paul & Venkatasubramanian, 2014; Goodfellow et al., 2014) and from an empirical perspective (Erhan et al., 2009; Eigen et al., 2013; Szegedy et al., 2013; Simonyan et al., 2013; Zeiler & Fergus, 2014; Nguyen et al., 2014; Yosinski et al., 2014; Mahendran &…”
Deeply-Recursive Convolutional Network for Image Super-Resolution (2015, background): “In Eigen et al. [6], recursive layers have the same input and output dimension, but recursive convolutions resulted in worse performances than a single convolution due to over-ﬁtting.”
Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex (2016, methodology): “The convolutional version was ﬁrst studied by [4].”
Do Semantic Parts Emerge in Convolutional Neural Networks? (2016, background): “2014), equivariance, invariance and equivalence (Lenc andVedaldi 2015), the ability to reconstruct the input (Mahendran and Vedaldi 2015) and how the number of layers, filters and parameters affects the network performance (Agrawal et al. 2014; Eigen et al. 2013).”
Deep architectures for modulation recognition (2017, methodology): “[2] suggests that the size of filters also has minimal impact, but based on expert knowledge of the radio domain and the dataset we expect 8tap filters to be optimal.”
Do Deep Convolutional Nets Really Need to be Deep and Convolutional? (2016, background): “A study of deep convolutional nets suggests that for vision tasks deeper models are preferred under a parameter budget (e.g. Eigen et al. (2014); He et al. (2015); Simonyan and Zisserman (2014); Srivastava et al. (2015)).”
Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? (2016, background): “power of a CNN lies in its deep architecture [5]–[8], which al-”
Automated diagnosis of prostate cancer in multi-parametric MRI based on multimodal convolutional neural networks (2017, background): “In recent years, deep convolutional neural networks (CNNs) have been widely demonstrated to be effective for various computer vision tasks when provided with a sufficient number of annotated data for training (Eigen et al 2013, Simonyan and Zisserman 2014, Szegedy et al 2015).”
SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability (2017, background): “Facets of this key question have been explored empirically, particularly for image models, in [1, 2, 10, 12, 13, 14, 15, 19, 20].”
Recursive Recurrent Nets with Attention Modeling for OCR in the Wild (2016, background): “Another way to expand longer data dependencies while controlling the model capacity is to make the Base CNN network recursive or recurrent as suggested in [35, 7, 29].”
SVCCA: Singular Vector Canonical Correlation Analysis for Deep Understanding and Improvement (2017, background): “Facets of this key question have been explored empirically, particularly for image models, in (1; 2; 8; 10; 11; 12; 13; 16; 17).”