Faster CNNs with Direct Sparse Convolutions and Guided Pruning

By Jongsoo Park, Sheng R. Li, Wujie Wen, Ping Tak Peter Tang, Hai Helen Li, Yiran Chen, Pradeep K. Dubey. 2016

189 citations recorded by the source corpus.

Phenomenally successful in practical inference problems, convolutional neural networks (CNN) are widely deployed in mobile devices, data centers, and even supercomputers. The number of parameters needed in CNNs, however, are often large and undesirable. Consequently, various methods have been developed to prune a CNN once it is trained. Nevertheless, the resulting CNNs offer limited benefits. While pruning the fully connected layers reduces a CNN's size considerably, it does not improve inference speed noticeably as the compute heavy parts lie in convolutions. Pruning CNNs in a way that increase inference speed often imposes specific sparsity structures, thus limiting the achievable sparsity levels. We present a method to realize simultaneously size economy and speed improvement while pruning CNNs. Paramount to our success is an efficient general sparse-with-dense matrix multiplication implementation that is applicable to convolution of feature maps with kernels of arbitrary sparsity patterns. Complementing this, we developed a performance model that predicts sweet spots of sparsity levels for different layers and on different computer architectures. Together, these two allow us to demonstrate 3.1--7.3$\times$ convolution speedups over dense convolution in AlexNet, on Intel Atom, Xeon, and Xeon Phi processors, spanning the spectrum from mobile devices to supercomputers. We also open source our project at this https URL.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “Structure pruning and growing were proposed for different goals, such as efficient inference [7, 8, 11, 13, 14, 18, 19, 24, 25, 27, 38–40], lifelong learning [41] and model adaptation [10, 29].”
Similarity-Preserving Knowledge Distillation (2019, background): “For example, unstructured pruning requires optimized sparse matrix multiplication routines to realize practical acceleration [26], platform constraint-aware compression [2, 36, 37] requires hardware simulators or empirical measurements, and arbitrarybit quantization [9, 17] requires specialized hard”
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning (2017, background): “From the perspective of inference acceleration, sparse and quantized Deep Neural Networks (DNNs) have been widely studied, such as [17][18][19][20][21][22][23][24][25].”
Sparse GPU Kernels for Deep Learning (2020, background): “[50] implement an efficient direct sparse convolution for CPUs and demonstrate performance gains relative to dense baselines.”
Interleaved Structured Sparse Convolutional Neural Networks (2018, background): “The efforts along this path mainly lie in how to perform optimization, and the technologies include non-structure sparsity regularization [24, 29], and structure sparsity regularization [39, 28].”
Jointly Sparse Convolutional Neural Networks in Dual Spatial-winograd Domains (2019, unknown): “, see [3, 14–18].”
Coordinating Filters for Faster Deep Neural Networks (2017, methodology): “The studies on sparse DNNs can be categorized into two types: nonstructured [20][23][22][8][6] and structured [28][21][19][1] sparsity methods.”
YOLOv4-5D: An Effective and Efficient Object Detector for Autonomous Driving (2021, methodology): “The pruning algorithms based on various strategies can reduce the number of network weights, computational complexity, and network redundancy [40].”
Accelerating Convolutional Networks via Global & Dynamic Filter Pruning (2018, background): “Under such a circumstance, specialized hardware [Han et al. , 2016] or software [Liu et al. , 2015; Park et al. , 2017] designs are required to further speedup the pruned unstructured CNNs.”
Comparing Rewinding and Fine-tuning in Neural Network Pruning (2020, background): “Unstructured pruning reduces the number of parameters, but may not improve performance on commodity hardware until a large fraction of weights have been pruned (Park et al., 2017).”
Distilling Object Detectors With Fine-Grained Feature Imitation (2019, background): “Connection pruning or weight sparsifying [14, 13, 27] prune redundant connections in large models.”
Accelerating Convolutional Neural Networks via Activation Map Compression (2018, methodology): “Pruning of weights [14, 18, 19, 34, 36, 40, 48, 63, 69] and quantization of weights and activations [6, 7, 12, 15, 18, 23, 50, 64, 67] are the standard compression techniques that are currently used.”
HRank: Filter Pruning Using High-Rank Feature Map (2020, background): “Besides network compression, weight pruning approaches can also achieve acceleration on specialized software [28] or hardware [8].”
Constraint-Aware Deep Neural Network Compression (2018, methodology): “Since conv1 is memory bandwidth dominant, pruning this layer gives low speedup but makes it harder to preserve the accuracy [15]; by default, we do not prune conv1 in the latency experiments.”
SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning (2020, background): “[30] further proposed to enable fine-grained pruning speedup on general-purpose platforms.”
Distilling Object Detectors with Task Adaptive Regularization (2020, background): “Weight quantization [29,32,9] refers to representing the weights with fewer bits, pruning approach [10,21,27] removes unimportant”
Inducing and Exploiting Activation Sparsity for Fast Neural Network Inference (2020, methodology): “Prior work has taken advantage of CSR for computing convolutions when the kernels are sparse, on both GPUs (Park et al., 2016b) and CPUs (Park et al., 2016a), where one has the luxury of being able to pre-compress the sparse kernels prior to inference with no performance overhead.”
DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures (2019, background): “Some works formulate the problem as a sparsity-inducing optimization problem, such as an `1 regularization [14, 22] that can be optimized using standard gradient-based algorithms, or an `0 regularization [11, 13] which requires stochastic approximation or special optimization techniques.”