Speeding up Convolutional Neural Networks with Low Rank Expansions

By Max Jaderberg, Andrea Vedaldi, Andrew Zisserman. 2014

1,565 citations recorded by the source corpus.

The focus of this paper is speeding up the application of convolutional neural networks. While delivering impressive results across a range of computer vision and machine learning tasks, these networks are computationally demanding, limiting their deployability. Convolutional layers generally consume the bulk of the processing time, and so in this work we present two simple schemes for drastically speeding up these layers. This is achieved by exploiting cross-channel or filter redundancy to construct a low rank basis of filters that are rank-1 in the spatial domain. Our methods are architecture agnostic, and can be easily applied to existing CPU and GPU convolutional frameworks for tuneable speedup performance. We demonstrate this with a real world network designed for scene text character recognition [15], showing a possible 2.5× speedup with no loss in accuracy, and 4.5× speedup with less than 1% drop in accuracy, still achieving state-of-the-art on standard benchmarks.

Subfields: Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

FitNets: Hints for Thin Deep Nets (2014, background): “A few works such as matrix factorization (Jaderberg et al., 2014; Denton et al., 2014) focus on speeding-up deep networks’ convolutional layers at the expense of slightly deteriorating their performance.”
Convolutional neural networks at constrained time cost (2014, background): “Deep CNNs not only have been continuously advancing the image classiﬁcation accuracy [14, 21, 24, 1, 9, 22, 23], but also play as generic feature extractors for various recognition tasks such as object detection [6, 9], semantic segmentation [6, 8], and image retrieval [14, 19].”
Sequence-Level Knowledge Distillation (2016, neutral): “Other approaches for compression involve low rank factorizations of weight matrices (Denton et al., 2013; Jaderberg et al., 2014; Lu et al., 2016; Prabhavalkar et al., 2016), sparsity-inducing regularizers (Murray and Chiang, 2015), binarization of weights (Courbariaux et al.”
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “Our method bears a similarity to decoupled convolution [44], which factorizes a depthwise convolution [78,36,21] to a column convolution and a row convolution.”
Aggregated Residual Transformations for Deep Neural Networks (2016, methodology): “Decomposition (at spatial [6, 18] and/or channel [6, 21, 16] level) is a widely adopted technique to reduce redundancy of deep convolutional networks and accelerate/compress them.”
Data-Efficient Structured Pruning via Submodular Optimization (2022, neutral): “, 2015), while another set of methods use low-rank approximation or tensor factorization (Denil et al., 2013; Sainath et al., 2013; Jaderberg et al., 2014; Denton et al., 2014; Lebedev et al., 2015; Kim et al., 2015; Zhang et al., 2016; Su et al., 2018).”
LoRA: Low-Rank Adaptation of Large Language Models (2021, background): “Some prior works even explicitly impose the low-rank constraint when training the original neural network, such as [17, 30, 35, 38, 39], however, to the best of our knowledge, none of these works considers low-rank update for adaptation to downstream tasks.”
The Efficiency Misnomer (2021, methodology): “As an example, tensor factorization is used as one of the common techniques to accelerate the matrix multiplication and used for making neural network models more efﬁcient (Zhang et al., 2015b; Jaderberg et al., 2014).”
Dynamic Neural Networks: A Survey (2021, background): “network pruning [28], weight quantization [29], knowledge distillation [30] and low-rank approximation [31].”
Squeeze-and-Excitation Networks (2017, background): “Cross-channel correlations are typically mapped as new combinations of features, either independently of spatial structure [6, 20] or jointly by using standard convolutional filters [24] with 1×1 convolutions.”
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications (2017, background): “Additionally various factorizations have been proposed to speed up pre-trained networks [14, 20].”
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices (2017, methodology): “5× model still achieves ∼13× actual speedup under comparable classification accuracy (the theoretical speedup is 18×), which is much faster than previous AlexNet-level models or speedup approaches such as [14, 16, 22, 42, 43, 38].”
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks (2016, neutral): “Matrix factorization has been used by [24,25].”
Semantic Image Segmentation via Deep Parsing Network (2015, background): “Note that convolutions in DPN can be further speeded up by low-rank decompositions [14] of the filters and model compressions [13].”
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design (2018, positive): “For example, tensor decomposition is widely used in early works [14,36,37] to accelerate the matrix multiplication.”
Pruning Filters for Efficient ConvNets (2016, background): “e proposed to approximate convolutional operations by representing the weight matrix as a low rank product of two smaller matrices without changing the original number of ﬁlters (Denil et al. (2013); Jaderberg et al. (2014); Zhang et al. (2015b;a); Tai et al. (2016); Ioannou et al. (2016)). Other ap”
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation (2014, background): “As opposed to [6], this approach adapts to the input distribution without the need to iterate through the data.”
Compressing Deep Convolutional Networks using Vector Quantization (2014, methodology): “In this work, instead of the traditional matrix factorization methods considered in (Denton et al., 2014; Jaderberg et al., 2014), we mainly consider a series of information theoretical vector quantization methods (Jegou et al., 2011; Chen et al., 2010) for compressing dense connected layers.”
Quantized Convolutional Neural Networks for Mobile Devices (2015, methodology): “In [7, 13, 32, 31, 18, 17], lowrank approximation or tensor decomposition is adopted to speed-up convolutional layers.”
Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware (2018, methodology): “Our work is also orthogonal and complementary to techniques that modify DNNs to speed up inference [21, 30, 42, 27, 15].”