Learning compact recurrent neural networks

By Zhiyun Lu, Vikas Sindhwani, Tara N. Sainath. 2016

88 citations recorded by the source corpus.

Recurrent neural networks (RNNs), including long short-term memory (LSTM) RNNs, have produced state-of-the-art results on a variety of speech recognition tasks. However, these models are often too large in size for deployment on mobile devices with memory and latency constraints. In this work, we study mechanisms for learning compact RNNs and LSTMs via low-rank factorizations and parameter sharing schemes. Our goal is to investigate redundancies in recurrent architectures where compression can be admitted without losing performance. A hybrid strategy of using structured matrices in the bottom layers and shared low-rank factors on the top layers is found to be particularly effective, reducing the parameters of a standard LSTM by 75%, at a small cost of 0.3% increase in WER, on a 2,000-hr English Voice Search task.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Sequence-Level Knowledge Distillation (2016, neutral): “Other approaches for compression involve low rank factorizations of weight matrices (Denton et al., 2014; Jaderberg et al., 2014; Lu et al., 2016; Prabhavalkar et al., 2016), sparsity-inducing regularizers (Murray and Chiang, 2015), binarization of weights (Courbariaux et al.”
Exploring Sparsity in Recurrent Neural Networks (2017, neutral): “Yet another advantage, when compared to methods that need re-training (Han et al., 2015), is that our pruning technique is part of training and needs (Lu et al., 2016).”
Compression of Neural Machine Translation Models via Pruning (2016, background): “The word embeddings are then fed as input into the main network, which consists of two multilayer RNNs ‘stuck together’ — an encoder for the source language and a decoder for the target language, each with their own weights.”
Recent progresses in deep learning based acoustic models (2017, methodology): “In [166], [167], a similar method was proposed for learning compact LSTMs via low-rank factorization and parameters sharing schemes.”
Multiscale Hierarchical Convolutional Networks (2017, methodology): “A parameter redundancy has also been observed in the ﬁ-nal fully connected layers used by number of neural network architectures, which contain most of the CNN parameters (Cheng et al., 2015; Lu et al., 2016).”
On-Device Neural Language Model Based Word Prediction (2018, methodology): “Compression through matrix factorization (Sainath et al., 2013; Xue et al., 2013; Nakkiran et al., 2015; Prabhavalkar et al., 2016; Lu et al., 2016) has shown promising results in model compression but has been applied to the tasks of automatic speech recognition.”
Learning Intrinsic Sparse Structures within Long Short-term Memory (2017, background): “…structures in original DNNs, approximating the original function of DNNs (Denil et al. (2013), Jaderberg et al. (2014), Hinton et al. (2015), Lu et al. (2016), Prabhavalkar et al. (2016), Molchanov et al. (2017)), and designing DNNs with inherently compact structures (Szegedy et al. (2015),…”
Neural Machine Translation: A Review (2019, background): “A straight-forward choice for score(·) proposed by Luong et al. (2015b) is the dot product (i.”
Soft Threshold Weight Reparameterization for Learnable Sparsity (2020, methodology): “Structured sparsity includes methods which make parameter tensors lowrank (Jaderberg et al., 2014; Lu et al., 2016; Alizadeh et al., 2019), prune out channels, filters and induce block/group sparsity (Liu et al.”
Separable Layers Enable Structured Efficient Linear Substitutions (2019, unknown): “• Toeplitz-like transforms (Lu et al., 2016; Sindhwani et al., 2015).”
Compression of Deep Learning Models for Text: A Survey (2020, background): “structured matrices [98] are highly specific and deterministic.”