GPU Kernels for Block-Sparse Weights

By Scott Gray, Alec Radford, Diederik P. Kingma. 2017

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Scaling Laws for Neural Language Models (2020, background): “Sparsity [CGRS19, GRK17] or branching (e.g. [KSH12]) may allow for even faster training of large networks through increased model parallelism.”
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (2021, background): “Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications.”
Big Bird: Transformers for Longer Sequences (2020, background): “well known [34, 103], that such sparse multiplications cannot be efficiently implemented in GPUs.”
Longformer: The Long-Document Transformer (2020, methodology): “The model with the most similar attention pattern to ours is Sparse Trans-former (Child et al., 2019), which uses a form of dilated sliding window of blocks of size 8x8 provided by BlockSparse (Gray et al., 2017).”
What is the State of Neural Network Pruning? (2020, neutral): “Kalchbrenner et al. (2018) obtain a similar result for audio synthesis, as do Gray et al. (2017) for a variety of additional tasks across various domains.”
Dota 2 with Large Scale Deep Reinforcement Learning (2019, methodology): “We use ops from the blocksparse library for fast GPU training[22].”
The hardware lottery (2020, background): “In parallel, there is interesting research developing specialized software kernels to support un-structured sparsity (Elsen et al., 2020; Gale et al., 2020; Gray et al., 2017).”
Blockwise Self-Attention for Long Document Understanding (2019, background): “of them have to represent a sparse matrix using a dense matrix with a binary mask or rely on customized CUDA kernels (Gray et al., 2017).”
Sparse Networks from Scratch: Faster Training without Losing Performance (2019, background): “ImageNet experiments were run on 4x RTX 2080 Ti and all other experiments on individual GPUs.”
Monarch: Expressive Structured Matrices for Efficient and Accurate Training (2022, background): “Our parameterization leverages optimized batch matrix multiply routines on GPUs, yielding up to 2× speedup compared to dense matrix multiply.”
Sparsely Aggregated Convolutional Networks (2018, background): “Others have demonstrated the benefits of a range of schemes [22,37,30,9,19] for sparsifying micro-architectural network structure (parameter structure within layers or filters).”
A Practical Survey on Faster and Lighter Transformers (2021, background): “Text Classiication SST2 Sparse byte mLSTM [39] BERT[24] 1.”
Accelerating Sparse Deep Neural Networks (2021, background): “Most prominent example of this category is block sparsity, where sparsity is introduced at a block level allowing good utilization of matrix math pipelines [32].”
Improving Language Understanding by Generative Pre-Training (2018, methodology): “Method Classification Semantic Similarity GLUE CoLA SST2 MRPC STSB QQP (mc) (acc) (F1) (pc) (F1) Sparse byte mLSTM [16] - 93.”
Inducing and Exploiting Activation Sparsity for Fast Neural Network Inference (2020, methodology): “Pruning and quantization are arguably the standard methods for achieving resource-efﬁcient models, which have received considerable attention, e.g. (Liu et al., 2017; Luo et al., 2017; Gray et al., 2017; Han et al., 2015; Li et al., 2016; Mishra et al., 2017; Zhu et al., 2016).”
Triton: an intermediate language and compiler for tiled neural network computations (2019, background): “Unfortunately, these libraries only support a restricted set of tensor operations, leaving the implementation of novel primitives to experts [13, 17, 25].”