Sparse GPU Kernels for Deep Learning

By Trevor Gale, Matei A. Zaharia, Cliff Young, Erich Elsen. 2020

296 citations recorded by the source corpus.

Scientific workloads have traditionally exploited high levels of sparsity to accelerate computation and reduce memory requirements. While deep neural networks can be made sparse, achieving practical speedups on GPUs is difficult because these applications have relatively moderate levels of sparsity that are not sufficient for existing sparse kernels to outperform their dense counterparts. In this work, we study sparse matrices from deep learning applications and identify favorable properties that can be exploited to accelerate computation. Based on these insights, we develop high-performance GPU kernels for two sparse matrix operations widely applicable in neural networks: sparse matrix–dense matrix multiplication and sampled dense– dense matrix multiplication. Our kernels reach 27% of single-precision peak on Nvidia V100 GPUs. Using our kernels, we demonstrate sparse Transformer and MobileNet models that achieve $1.2-2.1 \times $ speedups and up to $12.8 \times $ memory savings without sacrificing accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (2021, background): “Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications.”
Efficient Content-Based Sparse Attention with Routing Transformers (2020, background): “lack of support for sparse operations on the TPU; on the GPU various sparse kernels have been proposed which promise to significantly speed up training of these models (Gale et al., 2020).”
The hardware lottery (2020, background): “In parallel, there is interesting research developing specialized software kernels to support un-structured sparsity (Elsen et al., 2020; Gale et al., 2020; Gray et al., 2017).”
Layer-adaptive Sparsity for the Magnitude-based Pruning (2020, background): “While such unstructured pruning techniques have been considered less practical (compared to structured pruning), several recent breakthroughs provide promising methods to bridge this gap; see Gale et al. (2020); Elsen et al. (2020).”
nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models (2022, background): “Another issue with running GPUs for inference is that GPUs have prioritized high memory bandwidth over memory size [31], [32].”
Scatterbrain: Unifying Sparse and Low-rank Attention Approximation (2021, unknown): “This is the same limitation suffered by other sparse attention methods, but we are excited that more efficient sparse GPU kernels are being developed [31, 29].”
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation (2021, background): “The development of specialized software kernels has enabled the acceleration of sparse networks on traditional hardware (Gale et al., 2020; Elsen et al., 2019; Zhu et al., 2019) with new generations of hardware directly facilitating sparse training (Zhu et al., 2019).”