Block-Sparse Recurrent Neural Networks

By Sharan Narang, Eric Undersander, G. Diamos. 2017

147 citations recorded by the source corpus.

Recurrent Neural Networks (RNNs) are used in state-of-the-art models in domains such as speech recognition, machine translation, and language modelling. Sparsity is a technique to reduce compute and memory requirements of deep learning models. Sparse RNNs are easier to deploy on devices and high-end server processors. Even though sparse operations need less compute and memory relative to their dense counterparts, the speed-up observed by using sparse operations is less than expected on different hardware platforms. In order to address this issue, we investigate two different approaches to induce block sparsity in RNNs: pruning blocks of weights in a layer and using group lasso regularization with pruning to create blocks of weights with zeros. Using these techniques, we can create block-sparse RNNs with sparsity ranging from 80% to 90% with a small loss in accuracy. This technique allows us to reduce the model size by roughly 10x. Additionally, we can prune a larger dense network to recover this loss in accuracy while maintaining high block sparsity and reducing the overall parameter count. Our technique works with a variety of block sizes up to 32x32. Block-sparse RNNs eliminate overheads related to data storage and irregular memory accesses while increasing hardware efficiency compared to unstructured sparsity.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

MegaBlocks: Efficient Sparse Training with Mixture-of-Experts (2022, background): “Lastly, block-sparse kernels like matrix multiplication and convolution are general-purpose primitives that are useful across a range of applications (Narang et al., 2017; Gray et al., 2017; Child et al., 2019; Elsen et al., 2020).”
Efficient Neural Audio Synthesis (2018, methodology): “We use a pruning scheme based on the weight magnitude that increases sparsity as training proceeds (Narang et al., 2017a; Zhu & Gupta, 2017).”
Sparse GPU Kernels for Deep Learning (2020, background): “Despite model quality loss, it remains possible to exploit this hardware with vector and block sparsity [12], [13], [16].”
Fully Quantized Transformer for Machine Translation (2019, methodology): “With a similar objective, Narang et al. (2017b) make better use of hardware by applying pruning and weight decay in blocks to minimize the number of loaded weight matrix chunks.”
Triton: an intermediate language and compiler for tiled neural network computations (2019, background): “…convolutions (e.g., MobileNet [20]), they are often much slower than vendor libraries in practice (see, e.g., Figure 1), and lack the expressivity necessary to implement structured sparsity patterns [28, 31, 47] that cannot be directly specified using affine array indices in nested loops.”
Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM (2019, background): “In the domain of recurrent NNs, a relevant work explores hardware-inspired weight-and block-level sparsity for speech recognition [28].”
Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning (2020, methodology): “We compare our BSP method with irregular sparse format and the block sparse format (Narang et al., 2017; Gray et al., 2017) (pruning all weights on selected blocks).”
FeatherTTS: Robust and Efficient attention based Neural TTS (2020, methodology): “Inspired by [20, 21], we adopt the weight pruning scheme based on the weight magnitude.”
Compressing RNNs for IoT devices by 15-38x using Kronecker Products (2019, background): “…in neural network (NN) compression can be roughly categorized into 4 topics: pruning (Han et al., 2016; Zhu & Gupta, 2017; Changpinyo et al., 2017; Narang et al., 2017), structured matrix based techniques (Sindhwani et al., 2015; Ding et al., 2018; Wang et al., 2018; Ding et al., 2017),…”
Structured Pruning of Large Language Models (2019, background): “Earlier work focuses mostly on unstructured pruning, where weights are pruned individu-∗ ally (Narang et al., 2017a; Zhu and Gupta, 2017).”
Learned Token Pruning for Transformers (2021, methodology): “For structured pruning in NLP, [5, 32, 52] first explore the row and block-wise pruning for LSTM models [19].”
FeatherWave: An efficient high-fidelity neural vocoder with multi-band linear prediction (2020, methodology): “A block sparse pruning [17] strategy is adopted to sparsify the parameters in the GRU layer for the purpose of speeding up inference.”
GPU Kernels for Block-Sparse Weights (2017, methodology): “Closely related to our work, [Narang et al., 2017] used carefully scheduled thresholding and group lasso on blocks of weight matrices to learn block-sparse RNNs. [Wen et al., 2017] achieved unit level pruning via group lasso regularization of “Intrinsic Sparse Structure weight groups” - which…”
Chunked Autoregressive GAN for Conditional Waveform Synthesis (2021, methodology): “WaveRNN uses a smaller architecture and methods such as weight sparsiﬁcation (Narang et al., 2017; Zhu & Gupta, 2017) to improve generation speed without sacriﬁcing quality.”
A comprehensive survey on model compression and acceleration (2020, methodology): “Inspired from Narang et al. (2017), Narang et al. (2018) proposed a method which during the training of the network zeros the blocks of weights in the matrix using lasso regularization.”
MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Models (2021, background): “However, none of them considers diﬀerent sparse levels together or works on transformer-based models; and the algorithms underlying [28, 47, 6] are very diﬀerent than the pruning algorithm used in MLPruning.”
Balanced Sparsity for Efficient DNN Inference on GPU (2018, methodology): “…2017; Wen et al. 2016), gate and cell state sparsity for RNN (Wen et al. 2018), low rank approximation (Jader-berg, Vedaldi, and Zisserman 2014; Liu et al. 2015), and block sparsity (Narang, Undersander, and Diamos 2017) are several sparsity patterns in which model structure is fully considered.”