Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups

By Yani Ioannou, Duncan P. Robertson, Roberto Cipolla, Antonio Criminisi. 2016

295 citations recorded by the source corpus.

We propose a new method for creating computationally efficient and compact convolutional neural networks (CNNs) using a novel sparse connection structure that resembles a tree root. This allows a significant reduction in computational cost and number of parameters compared to state-of-the-art deep CNNs, without compromising accuracy, by exploiting the sparsity of inter-layer filter dependencies. We validate our approach by using it to train more efficient variants of state-of-the-art CNN architectures, evaluated on the CIFAR10 and ILSVRC datasets. Our results show similar or higher accuracy than the baseline architectures with much less computation, as measured by CPU and GPU timings. For example, for ResNet 50, our model has 40% fewer parameters, 45% fewer floating point operations, and is 31% (12%) faster on a CPU (GPU). For the deeper ResNet 200 our model has 48% fewer parameters and 27% fewer floating point operations, while maintaining state-of-the-art accuracy. For GoogLeNet, our model has 7% fewer parameters and is 21% (16%) faster on a CPU (GPU).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Aggregated Residual Transformations for Deep Neural Networks (2016, methodology): “Decomposition (at spatial [6, 18] and/or channel [6, 21, 16] level) is a widely adopted technique to reduce redundancy of deep convolutional networks and accelerate/compress them.”
Squeeze-and-Excitation Networks (2017, background): “org/challenges/LSVRC/2017/results dinality (the size of the set of transformations) [15, 47].”
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices (2017, methodology): “Group Convolution The concept of group convolution, which was first introduced in AlexNet [22] for distributing the model over two GPUs, has been well demonstrated its effectiveness in ResNeXt [41] and DeepRoots [15].”
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design (2018, background): “Group convolution is at the core of modern network architectures [33, 35, 12, 34, 31, 26].”
Efficient Structured Pruning and Architecture Searching for Group Convolution (2018, methodology): “To construct a CNN by GConv, one can build and train from scratch [50, 15, 45, 51, 43], or prune from pre-trained models.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “…efﬁcient NN model architectures : One line of work has focused on optimizing the NN model architecture in terms of its micro-architecture [100, 110, 125, 163, 164, 205, 245, 272] (e.g., kernel types such as depth-wise convolution or low-rank factorization) as well as its macro-architecture…”
Extreme Network Compression via Filter Group Approximation (2018, methodology): “To give a better understanding, we conduct the inter-layer filter correlation analyses [14].”
Moonshine: Distilling with Cheap Convolutions (2017, background): “…trained from scratch is actually on par with the original teacher (having a 0.12% lower top-1 error, and a 0.05% higher top-5 error) despite having 13 million fewer parameters; this generalisation capability of grouped convolutions in networks has been observed previously by Ioannou et al. (2017).”
DCFNet: Deep Neural Network with Decomposed Convolutional Filters (2018, background): “Sparse connections in CNNs have been recently studied in (Ioannou et al., 2016; Anwar et al., 2017; Changpinyo et al., 2017).”
Pruning neural networks: is it time to nip it in the bud? (2018, unknown): “The families of reduced networks used in this work are in no way exhaustive; incorporating grouped convolutions (Ioannou et al., 2017; Huang et al., 2018; Crowley et al., 2018) or dilated kernels (Yu & Koltun, 2016; Antoniou et al.”
SeerNet: Predicting Convolutional Neural Network Feature-Map Sparsity Through Low-Bit Quantization (2019, background): “Since model weights will not change after model training and are constant during inference, previous work proposes sparse matrix algorithms with statistical weight sparsity masks [10, 12, 19, 32, 8, 35] that achieve signiﬁcant model compression rate and inference speedup.”
Interleaved Structured Sparse Convolutional Neural Networks (2018, methodology): “Deep roots [11] instead uses the product of a structuredsparse kernel and a dense kernel.”
Building Efficient Deep Neural Networks With Unitary Group Convolutions (2018, background): “Deep Roots [10] uses group convolutions with increasing group size deeper into the network to improve numerous existing models.”
SBNet: Sparse Blocks Network for Fast Inference (2018, background): “Sparse computation in deep learning has been extensively explored in the weights domain, where the model size can be significantly reduced through pruning and lowrank decomposition [17, 27, 10, 32, 24, 14].”
DensePoint: Learning Densely Contextual Representation for Efficient Point Cloud Processing (2019, background): “The filter grouping can greatly reduce the model complexity, whilst leading to a model regularization by rarefying the filter relationships [15].”
Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection (2021, methodology): “In speciﬁc, the preprocessed data ﬁrst passes through a layer of 3 × 3 convolution block with three groups[18].”
Accelerating Convolutional Neural Networks via Activation Map Compression (2018, methodology): “The majority of the approaches [18, 26, 29] attempt to approximate the weights of a neural network in order to reduce the number of parameters (model compression) and/or the number of MAC’s (model acceleration).”
Two-branch Recurrent Network for Isolating Deepfakes in Videos (2020, methodology): “These maps are fused together with point-wise convolution with two groups [27] such that each group of convolutional filters independently refines and fuses the information for the three downstream DenseBlocks.”