Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition

By Hasim Sak, Andrew W. Senior, Franccoise Beaufays. 2014

1,120 citations recorded by the source corpus.

Long Short-Term Memory (LSTM) is a recurrent neural network (RNN) architecture that has been designed to address the vanishing and exploding gradient problems of conventional RNNs. Unlike feedforward neural networks, RNNs have cyclic connections making them powerful for modeling sequences. They have been successfully used for sequence labeling and sequence prediction tasks, such as handwriting recognition, language modeling, phonetic labeling of acoustic frames. However, in contrast to the deep neural networks, the use of RNNs in speech recognition has been limited to phone recognition in small scale tasks. In this paper, we present novel LSTM based RNN architectures which make more effective use of model parameters to train acoustic models for large vocabulary speech recognition. We train and compare LSTM, RNN and DNN models at various numbers of parameters and configurations. We show that LSTM models converge quickly and give state of the art speech recognition performance for relatively small sized models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Beyond human-level accuracy: computational challenges in deep learning (2019, methodology): “Since the prior word LM would take ~84 years per epoch to train, we start by choosing an algorithmic optimization used in recent word models: LSTM projection [30].”
On the State of the Art of Evaluation in Neural Language Models (2017, background): “Where the equations are based on the formulation of Sak et al. (2014).”
A Clockwork RNN (2014, background): “These networks have been very successful recently in speech and handwriting recognition (Graves et al., 2005; 2009; Sak et al., 2014).”
Training Production Language Models without Memorizing User Data (2020, methodology): “We use a single layer CIFG-LSTM [SSB14] neural network with shared weights between the input embedding layer and the output projection layer.”
Generative Models for Effective ML on Private, Decentralized Datasets (2019, methodology): “DP Word-LM The DP word-LM model is based on a coupled input forget gate CIFG-LSTM (Greff et al., 2015) with projection layer (Sak et al., 2014).”
Recent Trends in Deep Learning Based Natural Language Processing (2017, background): “This template is naturally suited for many NLP tasks such as language modeling [2, 64, 65], machine translation [66, 67, 68], speech recognition [69, 70, 71, 72], image captioning [73].”
Residual LSTM: Design of a Deep Recurrent Architecture for Distant Speech Recognition (2017, methodology): “Recurrent neural networks (RNN) were successfully applied to learn long term dependencies of sequential data [11, 12, 13, 14].”
Forecasting Economics and Financial Time Series: ARIMA vs. LSTM (2018, neutral): “…[Tarwani and Edem, 2017], handwriting recognition [Graves et al. 2009], speech recognition [Robinson, 2002; Eyben 2009; Graves et al. 2013; Sak et al. 2014], time-series prediction [Hochreiter and Schmidhuber, 1997; Gers, 2000; Yim, 2002; Zhang, 2005; Graves et al., 2009, 2013;…”
Sequence-to-sequence neural net models for grapheme-to-phoneme conversion (2015, methodology): “As in [5], we use an LSTM [17] as the basic recurrent network unit because it has shown better performance than simple RNNs on language understanding [24] and acoustic modeling [25] tasks.”
Exploring architectures, data and units for streaming end-to-end speech recognition with RNN-transducer (2017, background): “Deep recurrent neural networks with long short-term memory (LSTM) cells [2] have recently been shown to be ideal for this task [3, 4, 5].”
Spatio-temporal video autoencoder with differentiable memory (2015, methodology): “Architectures based on LSTM cells (Hochreiter & Schmidhuber, 1997) have been very successful in various tasks involving one-dimensional temporal sequences: speech recognition (Sak et al., 2014), machine translation (Sutskever et al.”
Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding (2018, background): “ ability to maintain memory of long-term dependencies due to the introduction of a weighted self-loop conditioned on context that allows them to forget past information in addition to accumulating it [17, 30, 37]. Their ability to handle high-complexity, temporal or sequential data has ensured their”
Machine Learning in Aerodynamic Shape Optimization (2022, background): “35) improves RNN by adding a forget gate (gf ), an update gate (gu), and an output gate (go) [260, 261].”
Context dependent phone models for LSTM RNN acoustic modelling (2015, methodology): “The LSTM networks are trained with a cross-entropy loss, using asynchronous stochastic gradient descent (ASGD) [4] using distributed training with 300 tasks scheduled on different machines, each working through a partition of the randomly shuffled training utterances.”
Modeling speaker variability using long short-term memory networks for speech recognition (2015, methodology): “Recently, literature [19] proposed a novel LSTM architecture, called LSTM Projected (LSTMP), which has a separate linear projection layer after the LSTM layer and outperforms the conventional LSTM networks.”
Advancing Multi-Accented Lstm-CTC Speech Recognition Using a Domain Specific Student-Teacher Learning Paradigm (2018, methodology): “Current successful ASR systems employ deep neural network (DNN) models as acoustic model combined with a hidden Markov model (HMM) [1, 2], or use them within an endto-end configuration [3, 4].”