Exploring Sparsity in Recurrent Neural Networks

By Nikhil Ketkar, Sharan Narang, G. Diamos, Shubho Sengupta, Erich Elsen. 2017

328 citations recorded by the source corpus.

Recurrent Neural Networks (RNN) are widely used to solve a variety of problems and as the quantity of data and the amount of available compute have increased, so have model sizes. The number of parameters in recent state-of-the-art networks makes them hard to deploy, especially on mobile phones and embedded devices. The challenge is due to both the size of the model and the time it takes to evaluate it. In order to deploy these RNNs efficiently, we propose a technique to reduce the parameters of a network by pruning weights during the initial training of the network. At the end of training, the parameters of the network are sparse while accuracy is still close to the original dense neural network. The network size is reduced by 8x and the time required to train the model remains constant. Additionally, we can prune a larger dense network to achieve better than baseline performance while still reducing the total number of parameters significantly. Pruning RNNs reduces the size of the model and can also help achieve significant inference time speed-up using sparse matrix multiply. Benchmarks show that using our technique model size can be reduced by 90% and speed-up is around 2x to 7x.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

The State of Sparsity in Deep Neural Networks (2019, background): “It has been shown empirically that deep neural networks can tolerate high levels of sparsity (Han et al., 2015; Narang et al., 2017; Ullrich et al., 2017), and this property has been leveraged to significantly reduce the cost associated with the deployment of deep neural networks, and to enable the…”
Efficient Neural Audio Synthesis (2018, methodology): “We use a pruning scheme based on the weight magnitude that increases sparsity as training proceeds (Narang et al., 2017a; Zhu & Gupta, 2017).”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “m connections. Srinivas et al. [47] and Louizos et al. [35] learn gating variables that minimize the number of nonzero network parameters. Narang et al. integrate magnitudebased pruning into training [40]. Gal and Ghahramani show that dropout approximates Bayesian inference in Gaussian processes [10”
A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, methodology): “Some recent work suggests [23] [48] that the network can achieve a sizable reduction in model size, yet achieving comparable accuracy.”
To prune, or not to prune: exploring the efficacy of pruning for model compression (2017, methodology): “A work similar to ours is the work by Narang et al. (2017) on pruning a RNN and GRU model for speech recognition and showing that a sparse RNN that was pruned outperformed a dense RNN trained normally of comparable size.”
Dynamic Model Pruning with Feedback (2020, background): “A pruning schedule determines when the new masks are computed (extending and simplifying (Narang et al., 2017)).”
Similarity-Preserving Knowledge Distillation (2019, methodology): “Weight pruning methods [11, 20, 22, 24, 33, 35, 39] remove unimportant weights from the network, sparsifying the network connectivity structure.”
Rigging the Lottery: Making All Tickets Winners (2019, methodology): “To overcome this problem, Narang et al. (2017) introduced gradual pruning, where connections are slowly removed over the course of a single round of training.”
Deep Rewiring: Training very sparse deep networks (2017, methodology): “A number of methods have been proposed that are capa-1 ble of reducing connectivity during training (Collins & Kohli, 2014; Jin et al., 2016; Narang et al., 2017).”
Sparse Networks from Scratch: Faster Training without Losing Performance (2019, methodology): “Extensions to this work include: Compressing recurrent neural networks and other models (Narang et al., 2017; Zhu and Gupta, 2018; Dai et al., 2018), continuous pruning and re-training (Guo et al., 2016), joint loss/pruning-cost optimization (Carreira-Perpinán and Idelbayev, 2018), layer-by-layer…”
Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization (2019, background): “Sparse reparameterization Successful training of sparse reparameterized networks usually employs iterative pruning and retraining, e.g. (Han et al., 2015b; Narang et al., 2017; Zhu & Gupta, 2017) 3.”
SNIP: Single-shot Network Pruning based on Connection Sensitivity (2018, background): “The magnitude criterion is also extended to recurrent neural networks (Narang et al. (2017)), yet with heavily tuned hyperparameter setting.”
Characterising Bias in Compressed Models (2020, background): “Despite the widespread use of compression techniques, articulating the trade-offs of compression has overwhelming centered on change to overall accuracy for a given level of compression (Ström, 1997; Cun et al., 1990; Evci et al., 2019; Narang et al., 2017).”
Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecture (2020, neutral): “, 2019) performs similar to magnitude pruning (Narang et al., 2017) on the transformer architecture (Vaswani et al.”
Fully Quantized Transformer for Machine Translation (2019, methodology): “With a similar objective, Narang et al. (2017b) make better use of hardware by applying pruning and weight decay in blocks to minimize the number of loaded weight matrix chunks.”
What Do Compressed Deep Neural Networks Forget (2019, methodology): “…model (model distillation) (Hinton et al., 2015) and ﬁnally pruning by setting a subset of weights or ﬁlters to zero (Louizos et al., 2017; Wen et al., 2016; Cun et al., 1990; Hassibi et al., 1993b; Ström, 1997; Hassibi et al., 1993a; Zhu & Gupta, 2017; See et al., 2016; Narang et al., 2017).”
Accelerating Sparse Deep Neural Networks (2021, background): “Sparse models can also be obtained by starting from randomly-initialized dense models and gradually introducing sparsity as training progresses [9,21,26–31].”
Model Pruning Enables Efficient Federated Learning on Edge Devices (2019, methodology): “In the iterative training and pruning approaches for the centralized machine learning setting, the model is first trained using SGD for a given number of iterations [13], [15], [16].”
Structured Pruning of Large Language Models (2019, background): “Earlier work focuses mostly on unstructured pruning, where weights are pruned individu-∗ ally (Narang et al., 2017a; Zhu and Gupta, 2017).”