Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
By Vadim Lebedev, Yaroslav Ganin, Maxim Rakhuba, Ivan V. Oseledets, Victor S. Lempitsky. 2014
952 citations recorded by the source corpus.
We propose a simple two-step approach for speeding up convolution layers within large convolutional neural networks based on tensor decomposition and discriminative fine-tuning. Given a layer, we use non-linear least squares to compute a low-rank CP-decomposition of the 4D convolution kernel tensor into a sum of a small number of rank-one tensors. At the second step, this decomposition is used to replace the original convolutional layer with a sequence of four convolutional layers with small kernels. After such replacement, the entire network is fine-tuned on the training data using standard backpropagation process. We evaluate this approach on two CNNs and show that it is competitive with previous approaches, leading to higher obtained CPU speedups at the cost of lower accuracy drops for the smaller of the two networks. Thus, for the 36-class character classification CNN, our approach obtains a 8.5x CPU speedup of the whole network with only minor accuracy drop (1% from 91% to 90%). For the standard ImageNet architecture (AlexNet), the approach speeds up the second convolution layer by a factor of 4x at the cost of $1\%$ increase of the overall top-5 classification error.
Subfields: Artificial Intelligence, Computational Mathematics, Computer Vision and Pattern Recognition, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computational Mathematics
- Timothy M. Hospedales
- Anima Anandkumar
- Yongxin Yang
- Jean Kossaifi
- Dmitry P. Vetrov
- Amnon Shashua
- Vivienne Sze
- Tien-Ju Yang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Ming-Hsuan Yang
- Yi Yang
- Trevor Darrell
- Alan L. Yuille
- Jitendra Malik
In Efficiency & Compression
- Song Han
- Ji Lin
- Yunhe Wang
- Jonathan Frankle
- Dan Alistarh
- Georgios Georgiadis
- Tijmen Blankevoort
- Dacheng Tao
Citations in the corpus
- Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “Some methods approximate model weights using quantization and hashing [Gong et al., 2014, Courbariaux et al., 2015], or low-rank approximation and tensor factorization [Denil et al., 2013, Lebedev et al., 2015, Su et al., 2018].”
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications (2017, background): “Additionally various factorizations have been proposed to speed up pretrained networks [14, 20].”
- ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices (2017, methodology): “5× model still achieves ∼13× actual speedup under comparable classification accuracy (the theoretical speedup is 18×), which is much faster than previous AlexNet-level models or speedup approaches such as [14, 16, 22, 42, 43, 38].”
- Quantized Convolutional Neural Networks for Mobile Devices (2015, methodology): “In Table 4, we report the performance under several ( C ′ s , K ) settings, comparing with two baseline methods, CPD [17] and GBD [18].”
- What is the State of Neural Network Pruning? (2020, background): “As others have noted (Lebedev et al., 2014; Figurnov et al., 2016; Louizos et al., 2017; Yang et al., 2017; Han et al., 2015; Kim et al., 2015; Wen et al., 2016; Luo et al., 2017; He et al., 2018b), these metrics are far from perfect.”
- A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, methodology): “Other similar work [2, 38, 6] to our work that reduces or removed the redundant connections or induces sparsity are motivated by the desire to speed up computation.”
- To prune, or not to prune: exploring the efficacy of pruning for model compression (2017, background): “…other potentially complementary approaches to reducing model size include low-rank matrix factorization (Denil et al., 2013; Denton et al., 2014; Jaderberg et al., 2014; Lebedev et al., 2014) and group sparsity regularization to arrive at an optimal layer size (Alvarez and Salzmann, 2016).”
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, background): “Tensor factorization [30, 15, 27, 35] decomposes weights into light-weight pieces, for example [51, 11, 14] proposed to accelerate the fully connected layers with truncated SVD; Jaderberg et al .”
- Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, methodology): “In contrast, tensor-wise examples include SVD decomposition [9,64], CPdecomposition [28], Tucker decomposition [22], Tensor-Train decomposition [45],”
- Compression of fully-connected layer in neural network by Kronecker product (2015, background): “…at reducing the number of parameters and amount of computation of neural network models, while keeping the quality of prediction results mostly the same.1 Model approximation is important for real world application of neural network to satisfy the time and storage constraints of the applications.”
- Compressing Convolutional Neural Networks in the Frequency Domain (2015, background): “Further speedup can be achieved by learning rank-one basis filters so that the convolution operations are very cheap to compute [14, 33].”
- Fast ConvNets Using Group-Wise Brain Damage (2015, methodology): “Alternatively, several recent works investigate various kinds of tensor factorization in order to break generalized convolution into a sequence of smaller convolutions with fewer parameters [21, 15, 29].”
- Accelerating Very Deep Convolutional Networks for Classification and Detection (2015, methodology): “These SGD-based solvers are sensitive to initialization and learning rates, and might be trapped into poorer local optima for regressing responses.”
- Channel Pruning for Accelerating Very Deep Neural Networks (2017, neutral): “Tensor factorization [22, 28, 13, 24] decompose weights into several pieces.”
- Dynamic Network Surgery for Efficient DNNs (2016, methodology): “Following similar ideas, some subsequent methods can provide more significant speedups [5, 22, 14].”
- Rethinking the Value of Network Pruning (2018, methodology): “To address this issue, many methods have been proposed such as low-rank approximation of weights (Denton et al., 2014; Lebedev et al., 2014), weight quantization (Courbariaux et al.”
- Deeper, Broader and Artier Domain Generalization (2017, methodology): “It is also related to lowrank neural network models, typically used to compress [13] and speed up [16] CNNs, and have very recently been explored for cross-category CNN knowledge transfer [31].”
- Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications (2015, methodology): “While (Lebedev et al., 2015; Zhang et al., 2015a) reported difficulty on finding a good SGD learning rate, our single learning rate scheduling rule works well for various compressed CNNs.”