Exploring the Regularity of Sparse Structure in Convolutional Neural Networks

By Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang, William (Bill) J. Dally. 2017

258 citations recorded by the source corpus.

Sparsity helps reduce the computational complexity of deep neural networks by skipping zeros. Taking advantage of sparsity is listed as a high priority in next generation DNN accelerators such as TPU. The structure of sparsity, i.e., the granularity of pruning, affects the efficiency of hardware accelerator design as well as the prediction accuracy. Coarse-grained pruning creates regular sparsity patterns, making it more amenable for hardware acceleration but more challenging to maintain the same accuracy. In this paper we quantitatively measure the trade-off between sparsity regularity and prediction accuracy, providing insights in how to maintain accuracy while having more a more structured sparsity pattern. Our experimental results show that coarse-grained pruning can achieve a sparsity ratio similar to unstructured pruning without loss of accuracy. Moreover, due to the index saving effect, coarse-grained pruning is able to obtain a better compression ratio than fine-grained sparsity at the same accuracy threshold. Based on the recent sparse convolutional neural network accelerator (SCNN), our experiments further demonstrate that coarse-grained sparsity saves about 2x the memory references compared to fine-grained sparsity. Since memory reference is more than two orders of magnitude more expensive than arithmetic operations, the regularity of sparse structure leads to more efficient hardware design.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Model compression has been explored from diﬀerent aspects [25, 37, 39, 34, 43, 20, 24, 50, 18, 74, 40, 26, 55, 59, 28, 60, 68, 33, 14, 38, 31].”
Efficient Structured Pruning and Architecture Searching for Group Convolution (2018, background): “, L1, L2 norms [27, 5, 20, 26], firstorder [29, 3] or second-order [19, 7, 2] gradients, average percentage of zero [26, 11], singular values [32, 28].”
Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks (2018, methodology): “, 2018b), `1 of filter weights (Mao et al., 2017; Li et al., 2017), `2 of filter weights and filter weights of the next layer (Jiang et al.”
CondenseNet: An Efficient DenseNet Using Learned Group Convolutions (2017, background): “For convolutional networks, different pruning techniques may lead to different levels of granularity [34].”
HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision (2019, methodology): “In [9, 23, 21], pruning is used to reduce the number of non-zero weights in NN models.”
I-BERT: Integer-only BERT Quantization (2021, neutral): “These techniques can be broadly categorized into (1) model pruning [18, 31, 35, 38, 40, 67], (2) knowledge distillation [21, 39, 43, 49, 70], (3) efficient neural architecture design [23, 24, 37, 51, 57], (4) hardware and neural architecture co-design [16, 17, 22, 29, 64], and (5) quantization [5, 7”
ZeroQ: A Novel Zero Shot Quantization Framework (2020, methodology): “These meth-ods are typically orthogonal to quantization, and they include efﬁcient neural architecture design [17, 9, 16, 36, 43], knowledge distillation [14, 35], model pruning [11, 29, 24], and hardware and NN co-design [9, 21].”
Interleaved Structured Sparse Convolutional Neural Networks (2018, unknown): “The efforts along this path mainly lie in how to perform optimization, and the technologies include non-structure sparsity regularization [24, 29], and structure sparsity regularization [39, 28].”
Efficient Processing of Deep Neural Networks: A Tutorial and Survey (2017, background): “These groups of weights can include a pair of neighboring weights, an entire row or column of a filter, an entire channel of a filter or the entire filter itself; using larger groups tends to result in higher loss in accuracy [150].”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, background): “The group of weights can be neighboring weights, part of a ﬁlter, or the whole ﬁlter [99].”
SnaPEA: Predictive Early Activation for Reducing Computation in Deep Convolutional Neural Networks (2018, background): “A handful of research [4], [6], [24]–[26] proposed various static pruning techniques to reduce the overhead of computation in deep convolutional neural networks.”
A review of convolutional neural network architectures and their optimizations (2022, background): “explored different levels of pruning granularities and found that the vector-level pruning occupies less memory space than the fine-grained pruning, since it requires fewer indices to indicate the pruning parameters (Mao et al. 2017).”
NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm (2017, unknown): “8 (2:0 ) Net pruning (ResNet-50) [39] Prune +0.”
Balanced Sparsity for Efficient DNN Inference on GPU (2018, methodology): “In our method, pruning followed by a retraining is one iteration, which is also deﬁned in previous methods (Han et al. 2015; Mao et al. 2017; Narang, Undersander, and Di-amos 2017).”
A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects (2020, background): “[167] deemed that coarse pruning reaches a better compression ratio than fine-grained pruning.”
Block-Sparse Recurrent Neural Networks (2017, methodology): “Mao et al. (2017) use a simple threshold based technique to create structurally sparse CNNs. Yu et al. (2017) propose Scalpel that prunes CNNs taking into account the underlying target hardware architecture.”
HAWQV3: Dyadic Neural Network Quantization (2020, background): “, 2019); (iii) Pruning redundant filters (Han et al., 2015; LeCun et al., 1990; Li et al., 2016; Mao et al., 2017; Molchanov et al., 2016; Yang et al., 2017); (iv) knowledge distillation (Hinton et al.”