Balanced Sparsity for Efficient DNN Inference on GPU

By Zhuliang Yao, Shijie Cao, Wencong Xiao, Chen Zhang, Lanshun Nie. 2018

99 citations recorded by the source corpus.

In trained deep neural networks, unstructured pruning can reduce redundant weights to lower storage cost. However, it requires the customization of hardwares to speed up practical inference. Another trend accelerates sparse model inference on general-purpose hardwares by adopting coarse-grained sparsity to prune or regularize consecutive weights for efficient computation. But this method often sacrifices model accuracy. In this paper, we propose a novel fine-grained sparsity approach, Balanced Sparsity, to achieve high model accuracy with commercial hardwares efficiently. Our approach adapts to high parallelism property of GPU, showing incredible potential for sparsity in the widely deployment of deep learning services. Experiment results show that Balanced Sparsity achieves up to 3.1x practical speedup for model inference on GPU, while retains the same high model accuracy as finegrained sparsity.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Ocean Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jonathan Frankle
2. Trevor Gale
3. Matei A. Zaharia
4. Cliff Young
5. Erich Elsen
6. John V. Guttag
7. Davis W. Blalock
8. Jose Javier Gonzalez Ortiz

In Computer Vision and Pattern Recognition

1. Jonathan Frankle
2. Matei A. Zaharia
3. Trevor Gale
4. Erich Elsen
5. Cliff Young
6. John V. Guttag
7. Davis W. Blalock
8. Jose Javier Gonzalez Ortiz

In Ocean Engineering

1. Jonathan Frankle
2. Asit K. Mishra
3. David S. Lawrie

Citations in the corpus

• What is the State of Neural Network Pruning? (2020, background): “In other cases, it means the fraction of parameters or FLOPs removed (Han et al., 2015; Lebedev & Lempitsky, 2016; Yao et al., 2018).”
• Sparse GPU Kernels for Deep Learning (2020, background): “[13] and [15] enforce different forms of structure on sparse matrices to enable efficient mapping to GPUs.”
• Structured Pruning of Large Language Models (2019, background): “…hand, structured pruning (Narang et al., 2017b; Wen et al., 2017; Cao et al., 2019; Yao et al., 2019) imposes highly structured sparse weight matrices that can either directly use optimized dense linear algebra primitives or admit efficient implementations (Gray et al., 2017; Yao et al., 2019).”
• FeatherWave: An efficient high-fidelity neural vocoder with multi-band linear prediction (2020, methodology): “%) usually degrades the model performance as mentioned in [20].”
• SeerNet: Predicting Convolutional Neural Network Feature-Map Sparsity Through Low-Bit Quantization (2019, background): “Since model weights will not change after model training and are constant during inference, previous work proposes sparse matrix algorithms with statistical weight sparsity masks [10, 12, 19, 32, 8, 35] that achieve significant model compression rate and inference speedup.”
• Accelerating Sparse Deep Neural Networks (2021, methodology): “Additionally, there has been work investigating fine-grained structured sparsity and motivating the need to prune in a fine-grained pattern that is conducive to hardware acceleration [17, 33].”