Coordinating Filters for Faster Deep Neural Networks

By Wujie Wen, Cong Xu, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Helen Li. 2017

143 citations recorded by the source corpus.

Very large-scale Deep Neural Networks (DNNs) have achieved remarkable successes in a large variety of computer vision tasks. However, the high computation intensity of DNNs makes it challenging to deploy these models on resource-limited systems. Some studies used low-rank approaches that approximate the filters by low-rank basis to accelerate the testing. Those works directly decomposed the pre-trained DNNs by Low-Rank Approximations (LRA). How to train DNNs toward lower-rank space for more efficient DNNs, however, remains as an open area. To solve the issue, in this work, we propose Force Regularization, which uses attractive forces to enforce filters so as to coordinate more weight information into lower-rank space1. We mathematically and empirically verify that after applying our technique, standard LRA methods can reconstruct filters using much lower basis and thus result in faster DNNs. The effectiveness of our approach is comprehensively evaluated in ResNets, AlexNet, and GoogLeNet. In AlexNet, for example, Force Regularization gains 2× speedup on modern GPU without accuracy loss and 4:05× speedup on CPU by paying small accuracy degradation. Moreover, Force Regularization better initializes the low-rank DNNs such that the fine-tuning can converge faster toward higher accuracy. The obtained lower-rank DNNs can be further sparsified, proving that Force Regularization can be integrated with state-of-the-art sparsity-based acceleration methods.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning to Prune Filters in Convolutional Neural Networks (2018, methodology): “Recently, [15] combined the low-rank approximation with channel pruning and [32] proposed to use Force Regularization to train neural networks towards low-rank spaces.”
AR-Net: Adaptive Frame Resolution for Efficient Action Recognition (2020, unknown): “Motivated by these applications, extensive studies have been recently conducted for designing compact architectures [44,27,28,64,2] or compressing models [13,57,9,35].”
Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification (2020, methodology): “Some other methods attempt to manipulate the “directions” of filters to implicitly reduce the rank of weight matrices [34, 20].”
Distilling Object Detectors With Fine-Grained Feature Imitation (2019, background): “Some works employ low rank approximation to large layers [33, 35].”
Interleaved Structured Sparse Convolutional Neural Networks (2018, background): “Channel pruning [25] and filter pruning [40, 22, 26] compute low-rank kernels in the output channel domain and the input channel domain, respectively1.”
NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES (2018, background): “Tensor factorization algorithms exploit the redundancy present in convolution layers by replacing the original tensors with a sequence of smaller or sparser counterparts that produce similar responses (Denton et al., 2014; Lebedev et al., 2014; Jaderberg et al., 2014; Cheng et al., 2016; Masana et a”
ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks (2018, background): “While compression is typically performed as a post-processing step, it has been shown that incorporating it during training could be beneficial [3, 4, 56, 57].”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “2) Tensorized Processing Engine: As mentioned above, most decomposed networks can be described as dense matrix operations with a smaller size, which are still suitable for running on general-purpose processors [139].”
A Practical Convolutional Neural Network as Loop Filter for Intra Frame (2018, unknown): “After that, filters are reconstructed using a much lower basis [15].”
Recent Advances in Convolutional Neural Network Acceleration (2018, methodology): “In the experiments, it obtains 2× faster on GPU without accuracy loss [57].”
Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer (2020, methodology): “Baseline 2 [47, 48] A simpler heuristic is to choose the lowest rank rk of matrix k, separately for each layer k, such that its rk-rank approximation error is within a 1 − p proportion of the norm of the original matrix:”
Compression-aware Training of Deep Networks (2017, background): “The contemporary work [51] introduced a penalty during training to learn correlated units.”
Review of Image Classification Algorithms Based on Convolutional Neural Networks (2021, methodology): “On the trained model: Singular Value Decomposition (SVD) [186] can achieve the effect of model compression by compressing the weight matrix of the fully connected layer in the network, Low-rank filter [187] uses two 1 × K conv instead of one K × K conv to remove redundancy and reduce weight paramete”
Learning Intrinsic Sparse Structures within Long Short-term Memory (2017, background): “Model Compression (Jaderberg et al. (2014), Han et al. (2015a), Wen et al. (2017), Louizos et al. (2017)) is a class of approaches to reduce the size of Deep Neural Networks (DNNs) and accelerate inference.”