Training CNNs with Low-Rank Filters for Efficient Image Classification

By Yani Ioannou, Duncan P. Robertson, Jamie Shotton, Roberto Cipolla, Antonio Criminisi. 2015

156 citations recorded by the source corpus.

We propose a new method for creating computationally efficient convolutional neural networks (CNNs) by using low-rank representations of convolutional filters. Rather than approximating filters in previously-trained networks with more efficient versions, we learn a set of small basis filters from scratch; during training, the network learns to combine these basis filters into more complex filters that are discriminative for image classification. To train such networks, a novel weight initialization scheme is used. This allows effective initialization of connection weights in convolutional layers composed of groups of differently-shaped filters. We validate our approach by applying it to several existing CNN architectures and training these networks from scratch using the CIFAR, ILSVRC and MIT Places datasets. Our results show similar or higher accuracy than conventional CNNs with much less compute. Applying our method to an improved version of VGG-11 network using global max-pooling, we achieve comparable validation accuracy using 41% less compute and only 24% of the original VGG-11 model parameters; another variant of our method gives a 1 percentage point increase in accuracy over our improved VGG-11 model, giving a top-5 center-crop validation accuracy of 89.7% while reducing computation by 16% relative to the original VGG-11 model. Applying our method to the GoogLeNet architecture for ILSVRC, we achieved comparable accuracy with 26% less compute and 41% fewer model parameters. Applying our method to a near state-of-the-art network for CIFAR, we achieved comparable accuracy with 46% less compute and 55% fewer parameters.

Subfields: Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Initialization and Regularization of Factorized Neural Layers (2021, background): “When these are smaller than the original matrix, the resulting networks are more efficient for both training and inference (Denil et al., 2013; Moczulski et al., 2015; Ioannou et al., 2016; Tai et al., 2016), resulting in model compression.”
Pruning Filters for Efficient ConvNets (2016, methodology): “…work have proposed to approximate convolutional operations by representing the weight matrix as a low rank product of two smaller matrices without changing the original number of ﬁlters (Denil et al. (2013); Jaderberg et al. (2014); Zhang et al. (2015b;a); Tai et al. (2016); Ioannou et al. (2016)).”
Spatiotemporal Residual Networks for Video Action Recognition (2016, methodology): “Based on the above observations, we developed a novel approach to temporal residual connections that builds on the ConvNet design guidelines of chaining small [21] asymmetric [10, 23] filters, noted in Sec.”
Learning Structured Sparsity in Deep Neural Networks (2016, methodology): “[13][12] improved and extended LRA to larger DNNs.”
Soft Weight-Sharing for Neural Network Compression (2017, background): “First introduced by (Denton et al., 2014) and Jaderberg et al. (2014), and elaborated on by using low rank ﬁlters (Ioannou et al., 2015), low rank regularization (Tai et al., 2015) or combining low rank decomposition with sparsity (Liu et al., 2015).”
Design of Efficient Convolutional Layers using Single Intra-channel Convolution, Topological Subdivisioning and Spatial "Bottleneck" Structure (2016, background): “Low rank decomposition has been extensively explored in various fashions [7][8][9][10][11] to obtain moderate efficiency improvement.”
Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups (2016, methodology): “Learning a combined spatial [9] and channel basis, may also improve efficiency further.”
BlockDrop: Dynamic Inference Paths in Residual Networks (2017, methodology): “The need to deploy top-performing deep neural network models on mobile devices motivates techniques that can effectively reduce the storage and computational costs of such networks, including knowledge distillation [19, 40, 4], low-rank factorization [25, 47, 41], filter pruning [30, 36, 32, 57], qu”
Fully-Adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification (2016, methodology): “It is complementary to existing task-agnostic approaches, such as knowledge distillation [12, 29], low-rank-factorization [14, 38, 32], pruning and quantization [10, 27], structured matrices [6, 34, 9], and dynamic capacity networks [2].”
Provable Filter Pruning for Efficient Neural Networks (2019, background): “, 2009) that compress a pre-trained network, to those approaches that enable sparsity by embedding sparsity as an objective directly in the training process (Ioannou et al., 2015; Alvarez & Salzmann, 2017) or exploit tensor structure to induce sparsity (Choromanska et al.”
Decomposable-Net: Scalable Low-Rank Compression for Neural Networks (2019, background): “(1) Before learning, [Ioannou et al., 2016] factorized a convolutional kernel into horizontal and vertical kernels.”
Recovering from Random Pruning: On the Plasticity of Deep Convolutional Neural Networks (2018, methodology): “Another approach is to use low rank factorization of the weight tensor/matrices to reduce the computations [21, 38, 22, 6, 40].”
A comprehensive survey on model compression and acceleration (2020, methodology): “After getting the promising model compression and acceleration result on the feed-forward NNs and CNNs, researchers tried to reduce the complexity of the RNNs. Reducing the word-length of weights and signals can help in reducing hardware complexity.”
Coordinating Filters for Faster Deep Neural Networks (2017, methodology): “Unlike sparsity-based methods, LowRank Approximation (LRA) methods [22][4][5][12][11] [26][27][18][30][14] directly decompose an original large model to a compact model with more lightweight layers.”
Recent Advances in Convolutional Neural Network Acceleration (2018, methodology): “Tai et al. utilize tensor decomposition to remove the redundancy in the convolution kernels, which achieves twice more efficiency of inference for VGG-16 [50].”
Factorized Convolutional Neural Networks (2016, background): “Low rank decomposition has been extensively explored in various fashions [3][10][25][8][22][11][24] to obtain moderate efficiency improvement.”
A Low Effort Approach to Structured CNN Design Using PCA (2018, background): “Authors of [23] learn a set of basis filters by introducing a new initialization technique.”
Accelerating Convolutional Neural Networks via Activation Map Compression (2018, methodology): “Other popular approaches include the modification of pre-trained networks by replacing the convolutional kernels with low-rank factorizations [25, 29, 56] or grouped convolutions [26].”