On the compression of recurrent neural networks with an application to LVCSR acoustic modeling for embedded speech recognition

By Rohit Prabhavalkar, Ouais Alsharif, Antoine Bruguier, Ian McGraw. 2016

104 citations recorded by the source corpus.

We study the problem of compressing recurrent neural networks (RNNs). In particular, we focus on the compression of RNN acoustic models, which are motivated by the goal of building compact and accurate speech recognition systems which can be run efficiently on mobile devices. In this work, we present a technique for general recurrent model compression that jointly compresses both recurrent and non-recurrent inter-layer weight matrices. We find that the proposed technique allows us to reduce the size of our Long Short-Term Memory (LSTM) acoustic model to a third of its original size with negligible loss in accuracy.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Sequence-Level Knowledge Distillation (2016, neutral): “Other approaches for compression involve low rank factorizations of weight matrices (Denton et al., 2013; Jaderberg et al., 2014; Lu et al., 2016; Prabhavalkar et al., 2016), sparsity-inducing regularizers (Murray and Chiang, 2015), binarization of weights (Courbariaux et al.”
Compression of Neural Machine Translation Models via Pruning (2016, background): “Based on that source representation, the decoder generates a translation, one target word at a time, and hence, decomposes the log conditional probability as: Most NMT work uses RNNs, but approaches differ in terms of: (a) architecture, which can be unidirectional, bidirectional, or deep multi-layer”
Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces (2018, methodology): “This experimental comparison, along with optional layer factorization (similar to [44]) or weight quantization are carried out for each target hardware setting, but this analysis is out of the scope of this paper.”
RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions (2020, background): “Each layer has 2048 units and a projection layer with 640 outputs units [26].”
Deep Learning in Mobile and Wireless Networking: A Survey (2018, unknown): “[249] Embedded speech recognition Edge-based LSTM Yoshioka et al.”
NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES (2018, methodology): “Within the structured pruning family, there are approaches based on the singular value decomposition (SVD) (Xue et al., 2013; Nakkiran et al., 2015; Prabhavalkar et al., 2016), where a new set of ﬁlters are obtained by projecting the original weights onto a lower dimensional space.”
Personalized speech recognition on mobile devices (2016, methodology): “LSTMs are made small and fast enough for embedded speech recognition by quantizing parameters to 8 bits, by using context independent (CI) phone outputs instead of more numerous context dependent (CD) phone outputs, and by using Singular Value Decomposition (SVD) compression [4, 5].”
Compression of Acoustic Event Detection Models With Quantized Distillation (2019, methodology): “Besides, low rank factorization [17, 18, 19, 20] is also commonly used in model compression, which estimates informative parameters by using matrix decomposition.”
Learning Intrinsic Sparse Structures within Long Short-term Memory (2017, background): “…in original DNNs, approximating the original function of DNNs (Denil et al. (2013), Jaderberg et al. (2014), Hinton et al. (2015), Lu et al. (2016), Prabhavalkar et al. (2016), Molchanov et al. (2017)), and designing DNNs with inherently compact structures (Szegedy et al. (2015), He et al. (2016),…”
Semi-Orthogonal Low-Rank Matrix Factorization for Deep Neural Networks (2018, methodology): “The most popular method to reduce the number of parameters of already-trained neural networks is to use Singular Value De-composition (SVD) to factorize each learned weight matrix as a product of two much smaller factors, discarding the smaller singular values [1, 2, 3].”
Compression of Deep Learning Models for Text: A Survey (2020, methodology): “[110] propose a method to jointly compress the recurrent and inter-layer matrices corresponding to a specific layer l by determining a suitable recurrent projection matrix, denoted by P l ∈ Rrl×Nl of rank r < N l such that W l h = Z l hP l and W x l = Z l xP .”
On-Device Neural Language Model Based Word Prediction (2018, methodology): “Compression through matrix factorization (Sainath et al., 2013; Xue et al., 2013; Nakkiran et al., 2015; Prabhavalkar et al., 2016; Lu et al., 2016) has shown promising results in model compression but has been applied to the tasks of automatic speech recognition.”
Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections (2016, background): “RNNs hold the state of the art result for automatic speech recognition (Prabhavalkar et al. (2016)).”
Recent progresses in deep learning based acoustic models (2017, methodology): “In [166], [167], a similar method was proposed for learning compact LSTMs via low-rank factorization and parameters sharing schemes.”
Streaming small-footprint keyword spotting using sequence-to-sequence models (2017, methodology): “The system is heavily compressed, both by quantization [43], and by the application of low-rank projection layers with 200 units between consecutive LSTM layers [44].”
Fully Neural Network Based Speech Recognition on Mobile and Embedded Devices (2018, methodology): “For efficient implementation of RNN, several model compression techniques have been developed, such as pruning [15], quantization [16], and matrix decomposition [17].”
State-of-the-Art Speech Recognition Using Multi-Stream Self-Attention with Dilated 1D Convolutions (2019, background): “Singular Value Decomposition (SVD) has been a popular choice to factorize a learned weight matrix into two low-rank factors and reduce the model complexity of neural networks [18, 19, 20].”
A comprehensive survey on model compression and acceleration (2020, methodology): “For RNN acoustic models, Prabhavalkar et al. (2016) proposed SVD-based factorization techniques for the compression of recurrent and non-recurrent (inter-layer) weight matrices.”