Learning Intrinsic Sparse Structures within Long Short-term Memory

By Wujie Wen, Yuxiong He, Samyam Rajbhandari, Minjia Zhang, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, Hai Helen Li. 2017

144 citations recorded by the source corpus.

Model compression is significant for the wide adoption of Recurrent Neural Networks (RNNs) in both user devices possessing limited resources and business clusters requiring quick responses to large-scale service requests. This work aims to learn structurally-sparse Long Short-Term Memory (LSTM) by reducing the sizes of basic structures within LSTM units, including input updates, gates, hidden states, cell states and outputs. Independently reducing the sizes of basic structures can result in inconsistent dimensions among them, and consequently, end up with invalid LSTM units. To overcome the problem, we propose Intrinsic Sparse Structures (ISS) in LSTMs. Removing a component of ISS will simultaneously decrease the sizes of all basic structures by one and thereby always maintain the dimension consistency. By learning ISS within LSTM units, the obtained LSTMs remain regular while having much smaller basic structures. Based on group Lasso regularization, our method achieves 10.59x speedup without losing any perplexity of a language modeling of Penn TreeBank dataset. It is also successfully evaluated through a compact model with only 2.69M weights for machine Question Answering of SQuAD dataset. Our approach is successfully extended to non- LSTM RNNs, like Recurrent Highway Networks (RHNs). Our source code is publicly available at this https URL

Subfields: Artificial Intelligence, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Efficiency & Compression

Citations in the corpus

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “Structure pruning and growing were proposed for different goals, such as efficient inference [7, 8, 11, 13, 14, 18, 19, 24, 25, 27, 38–40], lifelong learning [41] and model adaptation [10, 29].”
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning (2017, background): “From the perspective of inference acceleration, sparse and quantized Deep Neural Networks (DNNs) have been widely studied, such as [17][18][19][20][21][22][23][24][25].”
Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM (2019, methodology): “Follow-up works have successfully scaled this technique to LSTMs [5], [16], [17].”
Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware (2019, unknown): “Zhu, 2018; Frankle & Carbin, 2018), L0 and L1 Regularization (Louizos et al., 2017; Wen et al., 2017), Variational Dropout (Molchanov et al.”
Balanced Sparsity for Efficient DNN Inference on GPU (2018, background): “Filter and channel level sparsity for CNN (Li et al. 2016; Neklyudov et al. 2017; Wen et al. 2016), gate and cell state sparsity for RNN (Wen et al. 2018), low rank approximation (Jader-berg, Vedaldi, and Zisserman 2014; Liu et al. 2015), and block sparsity (Narang, Undersander, and Diamos 2017) are”
Learned Token Pruning for Transformers (2021, methodology): “For structured pruning in NLP, [5, 32, 52] first explore the row and block-wise pruning for LSTM models [19].”
DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures (2019, methodology): “[9, 15] present structural sparsity via group lasso, which applies an `1 regularization over the `2 norms of different groups of parameters.”
PruneTrain: Gradual Structured Pruning from Scratch for Faster Neural Network Training (2019, methodology): “Model pruning involves reducing the number of parameters in a dense fixed network (leading to lower memory and inference cost) while retaining the accuracy of the original model; it is already widely used and an active area of research (Wen et al., 2016; 2017; Feng & Darrell, 2015; He et al., 2018).”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “For example, the quantized networks can be viewed as dense models with shorter bit-width [179]; structurally pruned networks (e.g., vector-wise in RNNs and FM-wise in CNNs) [204], [205], [232], [234] or decomposed networks [139] can be reassembled into smaller dense models to speed up the running.”
Fully Quantized Transformer for Machine Translation (2019, background): “In order to maintain dimension consistency, Wen et al. (2017) propose to prune all basic LSTM structures concurrently.”
Compressing Gradient Optimizers via Count-Sketches (2019, methodology): “We hope to leverage recent ideas of adding sparsity to the hidden layers in order to increase the size of the model without increasing its computational cost [Shazeer et al. 2017; Spring and Shrivastava 2017; Wen et al. 2017].”
bert2BERT: Towards Reusable Pretrained Language Models (2021, background): “The module has three constraints: { ge out = g q|k|v in ; g q|k|v out = g o in; g q|k|v in = g o out}, with the first two constraints for hidden dimension consistency (Wen et al., 2018; Chen et al., 2021) and the third one for residual connection (Eq.”
MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Models (2021, background): “However, none of them considers diﬀerent sparse levels together or works on transformer-based models; and the algorithms underlying [28, 47, 6] are very diﬀerent than the pruning algorithm used in MLPruning.”
Structured Pruning of Large Language Models (2019, background): “Other works also attempt structured pruning but do not consider the l 0 regularization approach (Narang et al., 2017b; Wen et al., 2017; Cao et al., 2019; Yao et al., 2019).”
Block-Sparse Recurrent Neural Networks (2017, methodology): “Wen et al. (2017) alter the structure of Long Short Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) to create LSTMs with smaller memory footprint.”
Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification (2020, methodology): “This requires additional ﬁlter grouping during the pruning and ﬁlter rearranging after the pruning to make the pruned structure valid [32, 5].”
GPU Kernels for Block-Sparse Weights (2017, methodology): “[Wen et al., 2017] achieved unit level pruning via group lasso regularization of “Intrinsic Sparse Structure weight groups” - which correspond to the rows and columns of LSTM weight matrices for a specific unit.”