Deep Convolutional Networks are Hierarchical Kernel Machines

By Fabio Anselmi, Lorenzo Rosasco, Cheston Tan, Tomaso A. Poggio. 2015

54 citations recorded by the source corpus.

In i-theory a typical layer of a hierarchical architecture consists of HW modules pooling the dot products of the inputs to the layer with the transformations of a few templates under a group. Such layers include as special cases the convolutional layers of Deep Convolutional Networks (DCNs) as well as the non-convolutional layers (when the group contains only the identity). Rectifying nonlinearities -- which are used by present-day DCNs -- are one of the several nonlinearities admitted by i-theory for the HW module. We discuss here the equivalence between group averages of linear combinations of rectifying nonlinearities and an associated kernel. This property implies that present-day DCNs can be exactly equivalent to a hierarchy of kernel machines with pooling and non-pooling layers. Finally, we describe a conjecture for theoretically understanding hierarchies of such modules. A main consequence of the conjecture is that hierarchies of trained HW modules minimize memory requirements while computing a selective and invariant representation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Deriving Neural Architectures from Sequence and Graph Kernels (2017, neutral): “Similar results have been made for convolutional neural nets (Anselmi et al., 2015), and general computational graphs (Daniely et al.”
SGD Learns the Conjugate Kernel Class of the Network (2017, background): “, 2015, Bach, 2015, 2014, Hazan and Jaakkola, 2015, Anselmi et al., 2015] investigated such associations, but in a more restricted settings (i.e., for less architectures). Some of those papers [Rahimi and Recht, 2009, 2007, Daniely et al., 2016, Kar and Karnick, 2012, Bach, 2015, 2014] also provide ”
Training RNNs as Fast as CNNs (2017, background): “nd interpretability (Chen et al.,2018;Peng et al.,2018).Balduzzi and Ghifary(2016) applies type-preserving transformations to discuss the capacity of various simpliﬁed RNN architectures. Recent work (Anselmi et al., 2015;Daniely et al.,2016;Zhang et al.,2016;Lei et al.,2017) relates the capacity of ”
Group Invariance, Stability to Deformations, and Complexity of Deep Convolutional Representations (2017, unknown): “More recently, different models based on kernels have been introduced in [2, 14, 24] to gain some theoretical insight about classical multilayer neural networks, while kernels are used to define convex models for two-layer neural networks in [48].”
Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 2 Applications and Future Perspectives (2017, background): “However, the locality at each layer is a key factor which gives DCNNs an exponential advantage over shallow NNs in terms of the representation ability (Anselmi et al., 2015; Poggio et al., 2016; Mhaskar and Poggio, 2016).”
Neural and Computational Mechanisms of Action Processing: Interaction between Visual and Motor Representations. (2015, background): “Finally, some recent work also tries to develop a theoretical understanding of why hierarchical (deep) architectures have such favorable generalization properties (Anselmi et al., 2015).”
Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review (2016, background): “Very little appeared on multilayer networks, (but see [10, 11, 12] ), mainly because one hidden layer nets performed empirically as well as deeper networks.”
On the Expressive Power of Deep Learning: A Tensor Analysis (2015, unknown): “4 we take this further by showing that the shared HT model is exponentially more expressive than CP model, even if the latter is not constrained by sharing.”
Invariance and Stability of Deep Convolutional Representations (2017, unknown): “More recently, different models based on kernels are introduced in [2, 10, 18] to gain some theoretical insight about classical multilayer neural networks, while kernels are used to define convex models for two-layer neural networks in [36].”
Simple Recurrent Units for Highly Parallelizable Recurrence (2017, background): “Recent work (Anselmi et al., 2015; Daniely et al., 2016; Zhang et al., 2016; Lei et al., 2017) relates the capacity of neural networks to deep kernels.”