Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch
By Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, Hongsheng Li. 2021
313 citations recorded by the source corpus.
Sparsity in Deep Neural Networks (DNNs) has been widely studied to compress and accelerate the models on resource-constrained environments. It can be generally categorized into unstructured fine-grained sparsity that zeroes out multiple individual weights distributed across the neural network, and structured coarse-grained sparsity which prunes blocks of sub-networks of a neural network. Fine-grained sparsity can achieve a high compression ratio but is not hardware friendly and hence receives limited speed gains. On the other hand, coarse-grained sparsity cannot simultaneously achieve both apparent acceleration on modern GPUs and decent performance. In this paper, we are the first to study training from scratch an N:M fine-grained structured sparse network, which can maintain the advantages of both unstructured fine-grained sparsity and structured coarse-grained sparsity simultaneously on specifically designed GPUs. Specifically, a 2 : 4 sparse network could achieve 2× speed-up without performance drop on Nvidia A100 GPUs. Furthermore, we propose a novel and effective ingredient, sparse-refined straight-through estimator (SR-STE), to alleviate the negative influence of the approximated gradients computed by vanilla STE during optimization. We also define a metric, Sparse Architecture Divergence (SAD), to measure the sparse network’s topology change during the training process. Finally, We justify SR-STE’s advantages with SAD and demonstrate the effectiveness of SR-STE by performing comprehensive experiments on various tasks. Anonymous code and model will be at available at https://github.com/anonymous-NM-sparsity/NM-sparsity.
Subfields: Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Efficiency & Compression
Citations in the corpus
- Efficiently Scaling Transformer Inference (2022, neutral): “…2019), distillation (Sanh et al., 2019; Sun et al., 2020), and model compression techniques, such as pruning (Li et al., 2020b; Brix et al., 2020; Zhou et al., 2021; Li et al., 2020a; Wang et al., 2020), or quantization (Dettmers et al., 2022; Abdolrashidi et al., 2021; Zafrir et al., 2019;…”
- Sparse is Enough in Scaling Transformers (2021, methodology): “Structured pruning based approaches [47, 22, 43] account for this challenge by leveraging sparsity in hardware in CPU and GPU architectures [1].”
- Accelerating Sparse Deep Neural Networks (2021, background): “For this, one may need to train with dynamic sparsity masks, evolving them during training [65] (as opposed to static masks computed once during pruning, which this paper shows to suffice for inference).”
- SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (2023, methodology): “While so far we have only discussed unstructured pruning, SparseGPT is also easily adapted to semi-structured patterns such as the popular n:m sparsity format [49, 22] which delivers speedups in its 2:4 implementation on Ampere NVIDIA GPUs.”
- A Simple and Effective Pruning Approach for Large Language Models (2023, methodology): “Considering the potential need for practical speedup, we also conduct evaluations on structured N:M sparsity [31, 75].”
- Group Fisher Pruning for Practical Network Compression (2021, background): “Recently, there are also efforts (Zhou et al., 2021; Mishra et al., 2021) to develop N:M finegrained sparse models, leveraging the innovations in general-”
- Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (2022, unknown): “Finally, we note that the OBQ version of the techniques discussed in Section 4 has all the same runtime and memory characteristics (barring the global step in Figure 1, which is unnecessary for quantization).”