Long short-term memory recurrent neural network architectures for large scale acoustic modeling

By Hasim Sak, Andrew W. Senior, Franccoise Beaufays. 2014

2,571 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Factorization tricks for LSTM networks (2017, methodology): “We consider LSTM cell with projection of size p , LSTMP, where Equation 1 is computed as follows (Sak et al., 2014; Zaremba et al., 2014).”
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, methodology): “For MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014 Training: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters.”
Exploring the Limits of Language Modeling (2016, methodology): “…trying all combinations would be extremely time consuming for such a large data set, but our ﬁndings suggest that LSTMs with a projection layer (i.e., a bottleneck between hidden states as in (Sak et al., 2014)) trained with truncated BPTT (Williams & Peng, 1990) for 20 steps performed well.”
Dissecting Contextual Word Embeddings: Architecture and Representation (2018, methodology): “In particular, the LSTM with projection introduced by Sak et al. (2014) allows the model to use a large hidden state while reducing the total number of parameters.”
Semi-supervised sequence tagging with bidirectional language models (2017, methodology): “Following Sak et al. (2014), they used linear projection layers at the output of each LSTM layer to reduce the computation time but still maintain a large LSTM state.”
Unsupervised Pretraining for Sequence to Sequence Learning (2016, methodology): “, 2016) We used a 1 layer 4096 dimensional LSTM with the hidden state projected down to 1024 units (Sak et al., 2014) and trained for one week on 32 Tesla K40 GPUs.”
Toward Domain-Invariant Speech Recognition via Large Scale Training (2018, background): “The AM is a 5 layer unidirectional LSTM [23], with 1024 cells in the ﬁrst 4 layers, and 768 cells in the last layer.”
TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation (2018, methodology): “The RNNLM we retained was a mixture of three TDNN layers with two interspersed LSTMP layers [12] containing around 10 million parameters.”
Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, methodology): “End-to-end speech recognition is an active area of research, showing compelling results when used to re-score the outputs of a DNN-HMM [23] and standalone [26].”
Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss (2020, background): “the temporal dependencies in the audio features effectively [9] while maintaining a constant computational requirement for each frame.”
LSTM: A Search Space Odyssey (2015, background): “…and generation (Graves et al., 2013), language modeling (Zaremba et al., 2014) and translation (Luong et al., 2014), acoustic modeling of speech (Sak et al., 2014), speech synthesis (Fan et al., 2014), protein secondary structure prediction (Sønderby & Winther, 2014), analysis of audio (Marchi…”
Personalization of End-to-End Speech Recognition on Mobile Devices for Named Entities (2019, methodology): “The model consists of an encoder with 8 Long Short Term Memory (LSTM) [4] layers for the acoustic features, an encoder with 2 LSTM layers for the label sequences (denoted as the language model (LM) component) and a joint network with a single hidden layer.”
Federated Learning Of Out-Of-Vocabulary Words (2019, methodology): “In this work we use a variant of LSTM with a Coupled Input and Forget Gate (CIFG) (Greff et al., 2017), peephole connections (Gers and Schmidhuber, 2000) and a projection layer (Sak et al., 2014).”
The Limitations of Deep Learning in Adversarial Settings (2015, background): “In short, DL is transforming computational processing of data in many domains such as vision [24], [36], speech recognition [14], [32], language processing [12], financial fraud detection [23], and malware detection [13].”
Deep learning in neural networks: An overview (2014, neutral): “, 2014) and large-vocabulary speech recognition (Sak et al., 2014a).”
Smart Reply: Automated Response Suggestion for Email (2016, methodology): “In addition to the standard LSTM formulation, we found that the addition of a recurrent projection layer [18] substantially improved both the quality of the converged model and the time to converge.”
Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data (2016, background): “RNN models have been recently explored to capture notions of long and short term memory (LSTM) with the help of skip connections among nodes to model information delay [45].”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “and language applications, including handwriting recognition [37], generation of writings [38], language modeling and identiﬁcation [35, 146], automatic language translation [126], speech recognition [107, 29] analysis of audio data [78], analysis, annotation, and description of video data [22, 136,”