Accelerating Sparse Deep Neural Networks

By Asit K. Mishra, J. Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, Paulius Micikevicius. arXiv.org, 2021

344 citations recorded by the source corpus.

As neural network model sizes have dramatically increased, so has the interest in various techniques to reduce their parameter counts and accelerate their execution. An active area of research in this field is sparsity - encouraging zero values in parameters that can then be discarded from storage or computations. While most research focuses on high levels of sparsity, there are challenges in universally maintaining model accuracy as well as achieving significant speedups over modern matrix-math hardware. To make sparsity adoption practical, the NVIDIA Ampere GPU architecture introduces sparsity support in its matrix-math units, Tensor Cores. We present the design and behavior of Sparse Tensor Cores, which exploit a 2:4 (50%) sparsity pattern that leads to twice the math throughput of dense matrix units. We also describe a simple workflow for training networks that both satisfy 2:4 sparsity pattern requirements and maintain accuracy, verifying it on a wide range of common tasks and model architectures. This workflow makes it easy to prepare accurate models for efficient deployment on Sparse Tensor Cores.

Subfields: Computational Mathematics, Computer Vision and Pattern Recognition, Hardware and Architecture

View Paper: https://www.semanticscholar.org/paper/90d5e6f8d3b9f2617b3a3cf00fb02e730eb011cb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mathematics

In Computer Vision and Pattern Recognition

In Hardware and Architecture

Citations in the corpus

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (2023, methodology): “Second, as shown in Figure 1 (right), SparseGPT can also accurately impose sparsity in the more stringent, but hardware-friendly, 2:4 and 4:8 semi-structured sparsity patterns [32].”
Group Fisher Pruning for Practical Network Compression (2021, background): “Recently, there are also efforts (Zhou et al., 2021; Mishra et al., 2021) to develop N:M finegrained sparse models, leveraging the innovations in general-”
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (2022, neutral): “…approaches can be compounded, with surprisingly strong results: we obtain a 12 × reduction in theoretical operations with a 2% accuracy drop for GPU-supported compound compression [30], and a 4 × speedup in actual runtime with only 1% accuracy drop for a CPU-based sparsity-aware runtime [35].”
A Simple and Effective Pruning Approach for Large Language Models (2023, background): “While Wanda so far has been developed for unstructured sparsity, it can be easily extended to structured N:M sparsity (Mishra et al., 2021).”
How Well Do Sparse ImageNet Models Transfer? (2021, background): “at Thanks to this trend, a set of fairly consistent accuracy benchmarks has emerged for pruning, along with increasingly efficient computational support [11, 20, 40, 52].”
SliceGPT: Compress Large Language Models by Deleting Rows and Columns (2024, methodology): “Instead, we compare SliceGPT against SparseGPT (Frantar & Alistarh, 2023) employing a 2:4 sparsity ratio, as this is the only sparsity scheme which achieves speedup (Mishra et al., 2021).”