Factorization tricks for LSTM networks

By Oleksii Kuchaiev, Boris Ginsburg. 2017

122 citations recorded by the source corpus.

We present two simple ways of reducing the number of parameters and accelerating the training of large Long Short-Term Memory (LSTM) networks: the first one is "matrix factorization by design" of LSTM matrix into the product of two smaller matrices, and the second one is partitioning of LSTM matrix, its inputs and states into the independent groups. Both approaches allow us to train large LSTM networks significantly faster to the near state-of the art perplexity while using significantly less RNN parameters.

Subfields: Artificial Intelligence, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Signal Processing

Citations in the corpus

Attention is All you Need (2017, background): “Recent work has achieved significant improvements in computational efficiency through factorization tricks [18] and conditional computation [26], while also improving model performance in case of the latter.”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “…Inan et al. (2016); Press & Wolf (2016); Merity et al. (2017); Gal & Ghahramani (2016), speeding up the Softmax computation (Morin & Bengio, 2005; Kuchaiev & Ginsburg, 2017; Grave et al., 2016a; Jozefowicz et al., 2016), and enriching the output distribution family (Yang et al., 2017; Kanai et…”
Language Modeling with Gated Convolutional Networks (2016, neutral): “…2013) 51.3 24 GPUs LSTM-2048-512 (Jozefowicz et al., 2016) 43.7 32 GPUs 2-layer LSTM-8192-1024 (Jozefowicz et al., 2016) 30.6 32 GPUs BIG GLSTM-G4 (Kuchaiev & Ginsburg, 2017) 23.3 ∗ 8 GPUs LSTM-2048 (Grave et al., 2016a) 43.9 1 GPU 2-layer LSTM-2048 (Grave et al., 2016a) 39.8 1 GPU GCNN-13 38.1 1…”
Depthwise Separable Convolutions for Neural Machine Translation (2017, methodology): “We use the idea from group convolutions and the recent separable-LSTM paper [12] to further reduce this size by factoring the final 1× 1 convolution, and we call the result a super-separable convolution.”
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices (2020, background): “In the NLP literature, the most relevant work can be group LSTMs (Kuchaiev and Ginsburg, 2017; Gao et al., 2018), which employs the idea of group convolution (Zhang et al., 2017, 2018) into Recurrent Neural Networks (RNN).”
Federated Learning of N-Gram Language Models (2019, methodology): “For 4K word-piece models, GLSTM is in general on-par with its P 4K-L counterpart.”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, background): “Factorized LSTM [53] fac-torizes each weight matrix of an LSTM into two smaller matrices and feeds them with two subvectors.”
GPU Kernels for Block-Sparse Weights (2017, methodology): “This work has been further improved by [Kuchaiev and Ginsburg, 2017] who explore additional factorization methods for LSTMs.”
Exploring Interpretable LSTM Neural Networks over Multi-Variable Data (2019, background): “Kuchaiev & Ginsburg (2017); Neil et al. (2016); Koutnik et al. (2014) proposed to partition the hidden layer into separated modules with different updates.”
Transformer-XL: Language Modeling with Longer-Term Dependency (2018, background): “…Inan et al. (2016); Press & Wolf (2016); Merity et al. (2017); Gal & Ghahramani (2016), speeding up the Softmax computation (Morin & Bengio, 2005; Kuchaiev & Ginsburg, 2017; Grave et al., 2016a; Jozefowicz et al., 2016), and enriching the output distribution family (Yang et al., 2017; Kanai et…”
Compressing RNNs for IoT devices by 15-38x using Kronecker Products (2019, methodology): “Hyperparameters : Table 6 shows the hyperparameters used for training the MNIST-LSTM baseline and the MNIST network compressed using pruning, LMF, KPLSTM and a smaller baseline with the number of parameters equivalent to the compressed network.”
Simple Recurrent Units for Highly Parallelizable Recurrence (2017, methodology): “, 2017) and to speed up recurrent networks (Diamos et al., 2016; Shazeer et al., 2017; Kuchaiev and Ginsburg, 2017).”
Scale down Transformer by Grouping Features for a Lightweight Character-level Language Model (2020, unknown): “(Kuchaiev and Ginsburg, 2017) proposed group-wise RNN as a special form of ensembled RNNs.”
Training RNNs as Fast as CNNs (2017, methodology): “ned version of diagonal initialization. Various strategies have been proposed to scale network training (Goyal et al.,2017) and to speed up recurrent networks (Diamos et al.,2016; Shazeer et al.,2017;Kuchaiev and Ginsburg, 2017). For instance,Diamos et al.(2016) utilize hardware infrastructures by s”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, background): “The weight matrix of LSTM can be factorized into matrices, and results can be concatenated for faster convergence and parameter reduction [90].”