Convolutional neural networks with low-rank regularization

By Cheng Tai, Tong Xiao, Xiaogang Wang, E. Weinan. 2015

494 citations recorded by the source corpus.

Large CNNs have delivered impressive performance in various computer vision applications. But the storage and computation requirements make it problematic for deploying these models on mobile devices. Recently, tensor decompositions have been used for speeding up CNNs. In this paper, we further develop the tensor decomposition technique. We propose a new algorithm for computing the low-rank tensor decomposition for removing the redundancy in the convolution kernels. The algorithm finds the exact global optimizer of the decomposition and is more effective than iterative methods. Based on the decomposition, we further propose a new method for training low-rank constrained CNNs from scratch. Interestingly, while achieving a significant speedup, sometimes the low-rank constrained CNNs delivers significantly better performance than their non-constrained counterparts. On the CIFAR-10 dataset, the proposed low-rank NIN model achieves $91.31\%$ accuracy (without data augmentation), which also improves upon state-of-the-art result. We evaluated the proposed method on CIFAR-10 and ILSVRC12 datasets for a variety of modern CNNs, including AlexNet, NIN, VGG and GoogleNet with success. For example, the forward time of VGG-16 is reduced by half while the performance is still comparable. Empirical success suggests that low-rank tensor decompositions can be a very useful tool for speeding up large CNNs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Initialization and Regularization of Factorized Neural Layers (2021, methodology): “However, as shown by Tai et al. (2016), applying the 2d k ˆ k convolution with c i ´ 1 input channels and c i output channels obtained by reshaping UV T is equivalent to a composition of two 1d convolutions: the ﬁrst deﬁned by V T P R r ˆ c i ´ 1 k consists of r output channels and ﬁlters of size k…”
Pruning Filters for Efficient ConvNets (2016, methodology): “…work have proposed to approximate convolutional operations by representing the weight matrix as a low rank product of two smaller matrices without changing the original number of ﬁlters (Denil et al. (2013); Jaderberg et al. (2014); Zhang et al. (2015b;a); Tai et al. (2016); Ioannou et al. (2016)).”
RotDCF: Decomposition of Convolutional Filters for Rotation-Equivariant Deep Networks (2018, background): “[35] proposed low-rank-regularized ﬁlters and obtained improved classiﬁcation accuracy with reduced computation.”
Learning Structured Sparsity in Deep Neural Networks (2016, methodology): “[13][12] improved and extended LRA to larger DNNs.”
Soft Weight-Sharing for Neural Network Compression (2017, background): “, 2015), low rank regularization (Tai et al., 2015) or combining low rank decomposition with sparsity (Liu et al.”
Improved Knowledge Distillation via Teacher Assistant (2019, background): “Model compression techniques have emerged to address such issues, e.g. , parameter pruning and sharing (Han, Mao, and Dally 2016), low-rank factorization (Tai et al. 2015) and knowledge distillation (Bucila, Caruana, and Niculescu-Mizil 2006; Hinton, Vinyals, and Dean 2015).”
Design of Efficient Convolutional Layers using Single Intra-channel Convolution, Topological Subdivisioning and Spatial "Bottleneck" Structure (2016, background): “Low rank decomposition has been extensively explored in various fashions [7][8][9][10][11] to obtain moderate efﬁciency improvement.”
Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks (2018, methodology): “In particular, the matrix decomposition of deep CNN tensors is approximated by the product of two low-rank matrices [Jaderberg et al. , 2014; Zhang et al. , 2016; Tai et al. , 2016].”
Fully-Adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification (2016, methodology): “Methods for compressing and accelerating convolutional networks include knowledge distillation [12, 29], low-rank-factorization [14, 39, 33], pruning and quantization [10, 27], structured matrices [6, 35, 9], and dynamic capacity networks [2].”
Fix your classifier: the marginal value of training the last weight layer (2018, methodology): “Tai et al. (2015) and Jaderberg et al. (2014) used low rank approximations to speed up NNs. Hubara et al. (2016b), Li et al. (2016) and Zhou et al. (2016), used a more aggressive approach, in which weights, activations and gradients were quantized to further reduce computation during training.”
Factorized Higher-Order CNNs With an Application to Spatio-Temporal Emotion Estimation (2019, background): “Tensor decompositions (and, in particular, decomposed convolutions) are notoriously hard to train end-to-end [12, 27, 46].”
Learning with Hyperspherical Uniformity (2021, background): “There are quite a number of relational regularizations that have been used in neural networks, such as orthogonality regularization [4, 52, 67, 32, 15], unitary constraint [34, 81, 3], decorrelation [67, 16, 85], spectral regularization [86], low-rank regularization [76], angular constraint [84, 44]”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, unknown): “D matrix [78].”
Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon (2017, methodology): “Besides Net-trim, which is a layer-wise pruning method discussed in the previous section, there is some other work proposed to induce sparsity or low-rank approximation on certain layers for pruning [19, 20].”
Faster CNNs with Direct Sparse Convolutions and Guided Pruning (2016, methodology): “ Han et al. (2016b), Liu et al. (2015), Guo et al. (2016), GESL C: Denton et al. (2014), Jaderberg et al. (2014), Lebedev et al. (2015), Zhang et al. (2015), Kim et al. (2016), Ioannou et al. (2016), Tai et al. (2016), Denil et al. (2013) Pruning Regularization Factorization Group-wise Element-wise ”
Adversarially Robust Distillation (2019, background): “Such applications require efficient networks with light-weight inference costs (Dziugaite, Ghahramani, and Roy 2016; Sandler et al. 2018; Courbariaux et al. 2016; Tai et al. 2015).”
Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges (2018, methodology): “Baseline Models Representative Works Alexnet [1] structural matrix [30]–[32] low-rank factorization [39] Network in network [68] low-rank factorization [39] VGG nets [69] transfered filters [43] low-rank factorization [39] Residual networks [70] compact filters [48], stochastic depth [61] parameter ”
Decomposable-Net: Scalable Low-Rank Compression for Neural Networks (2019, methodology): “Note that our methods do not rely on a specific decomposition form, and the SVD-based methods, including channel- and spatial-wise [Zhang et al., 2016; Tai et al., 2016] decompositions, can be adopted.”
Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference (2020, methodology): “Layer factorization (Tai et al., 2016; Yu et al., 2017), quantization (Wu et al., 2016), model distillation (Wang et al., 2018c) and weight sharing (Wu et al., 2018) have also been respectively found effective.”