Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation

By Emily L. Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, Rob Fergus. 2014

1,841 citations recorded by the source corpus.

We present techniques for speeding up the test-time evaluation of large convolutional networks, designed for object recognition tasks. These models deliver impressive accuracy, but each image evaluation requires millions of floating point operations, making their deployment on smartphones and Internet-scale clusters problematic. The computation is dominated by the convolution operations in the lower layers of the model. We exploit the redundancy present within the convolutional filters to derive approximations that significantly reduce the required computation. Using large state-of-the-art models, we demonstrate speedups of convolutional layers on both CPU and GPU by a factor of 2 x, while keeping the accuracy within 1% of the original model.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Fast R-CNN (2015, background): “Recognizing this fact, we look to truncated SVD, which offers a simple way to compress fully-connected layers [5, 22].”
FitNets: Hints for Thin Deep Nets (2014, background): “A few works such as matrix factorization (Jaderberg et al., 2014; Denton et al., 2014) focus on speeding-up deep networks’ convolutional layers at the expense of slightly deteriorating their performance.”
Convolutional neural networks at constrained time cost (2014, background): “Deep CNNs not only have been continuously advancing the image classiﬁcation accuracy [14, 21, 24, 1, 9, 22, 23], but also play as generic feature extractors for various recognition tasks such as object detection [6, 9], semantic segmentation [6, 8], and image retrieval [14, 19].”
Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition (2014, methodology): “To our knowledge, our method is the first one that trains a single network with input images of multiple sizes.”
Sequence-Level Knowledge Distillation (2016, methodology): “Other approaches for compression involve low rank factorizations of weight matrices (Denton et al., 2014; Jaderberg et al., 2014; Lu et al., 2016; Prabhavalkar et al., 2016), sparsity-inducing regularizers (Murray and Chiang, 2015), binarization of weights (Courbariaux et al., 2016; Lin et al.,…”
Aggregated Residual Transformations for Deep Neural Networks (2016, methodology): “Decomposition (at spatial [6, 18] and/or channel [6, 21, 16] level) is a widely adopted technique to reduce redundancy of deep convolutional networks and accelerate/compress them.”
Data-Efficient Structured Pruning via Submodular Optimization (2022, neutral): “, 2015), while another set of methods use low-rank approximation or tensor factorization (Denil et al., 2013; Sainath et al., 2013; Jaderberg et al., 2014; Denton et al., 2014; Lebedev et al., 2015; Kim et al., 2015; Zhang et al., 2016; Su et al., 2018).”
Speeding up Convolutional Neural Networks with Low Rank Expansions (2014, methodology): “We also don’t approximate Conv1 due to the fact that it acts on raw pixels from natural images – the filters in Conv1 are very different to those found in the rest of the network and experimentally we found that they cannot be approximated well by separable filters (also observed in [7]).”
Learning both Weights and Connections for Efficient Neural Network (2015, methodology): “In 1998 Lecun et al. designed a CNN model LeNet-5 with less than 1M parameters to classify handwritten digits [4], while in 2012, Krizhevsky et al. [1] won the ImageNet competition with 60M parameters.”
Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding (2015, background): “For pruned AlexNet, we are able to quantize to 8-bits (256 shared weights) for each CONV layers, and 5-bits (32 shared weights) for each FC layer without any loss of accuracy.”
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size (2016, methodology): “The SVD-based approach 5Our baseline is bvlc alexnet from the Caffe codebase [19]. is able to compress a pretrained AlexNet model by a factor of 5X, while diminishing top-1 accuracy to 56.0% [5].”
The Loss Surfaces of Multilayer Networks (2014, background): “It was recently shown [Denil et al., 2013, Denton et al., 2014] that for large-size networks large number of network parameters (according to [Denil et al., 2013] even up to 95”
Compressing Deep Convolutional Networks using Vector Quantization (2014, methodology): “In particular, SVD achieves impressive results for compressing and speeding up convolutional layers (Denton et al., 2014) but does not work well for compressing dense connected layers.”
Compressing Neural Networks with the Hashing Trick (2015, methodology): “A follow-up work by Denton et al. (2014) uses a similar technique to speed up test-time evaluation of convolutional neural networks.”
Quantized Convolutional Neural Networks for Mobile Devices (2015, methodology): “Matrix factorization was adopted in [6, 7] to decompose the weighting matrix into two low-rank matrices, which demonstrated that significant redundancy did exist in network parameters.”
A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, methodology): “Besides quantization, other potentially complementary approaches to reducing model size include low-rank matrix factorization [8, 38] and group sparsity regularization to arrive at an optimal layer size [1].”
Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware (2018, methodology): “Our work is also orthogonal and complementary to techniques that modify DNNs to speed up inference [21, 30, 42, 27, 15].”
To prune, or not to prune: exploring the efficacy of pruning for model compression (2017, methodology): “Besides quantization, other potentially complementary approaches to reducing model size include low-rank matrix factorization (Denil et al., 2013; Denton et al., 2014; Jaderberg et al., 2014; Lebedev et al., 2014) and group sparsity regularization to arrive at an optimal layer size (Alvarez and…”
Similarity-Preserving Knowledge Distillation (2019, methodology): “Lowrank factorization methods [6, 7, 16, 27, 45] produce compact low-rank approximations of filter matrices.”