Fast and accurate recurrent neural network acoustic models for speech recognition

By Hasim Sak, Andrew W. Senior, Kanishka Rao, Franccoise Beaufays. 2015

443 citations recorded by the source corpus.

We have recently shown that deep Long Short-Term Memory (LSTM) recurrent neural networks (RNNs) outperform feed forward deep neural networks (DNNs) as acoustic models for speech recognition. More recently, we have shown that the performance of sequence trained context dependent (CD) hidden Markov model (HMM) acoustic models using such LSTM RNNs can be equaled by sequence trained phone models initialized with connectionist temporal classification (CTC). In this paper, we present techniques that further improve performance of LSTM RNN acoustic models for large vocabulary speech recognition. We show that frame stacking and reduced frame rate lead to more accurate models and faster decoding. CD phone modeling leads to further improvements. We also present initial results for LSTM RNN models outputting words directly.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

In Signal Processing

Citations in the corpus

BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition (2021, background): “The fusion weight [71] and the non-blank reward [83], [84] are selected by a small random exploration.”
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (2020, methodology): “Following [8], we re-tune the LM fusion parameters (the LM fusion weight [51] and the non-blank reward [53, 54]) at each generation to maximize dev set performance of the fused model.”
Multilingual Speech Recognition with a Single End-to-End Model (2017, methodology): “Following [19] we stack 8 consecutive frames and stride the stacked frames by a factor of 3.”
Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, methodology): “One possible solution is to delay the system from emitting a label until it has more context as in (Sak et al., 2015), but we found it difficult to induce this behavior in our models.”
Improved Noisy Student Training for Automatic Speech Recognition (2020, methodology): “We introduce a coverage penalty term [32] in the fusion score with parameter c for LAS models [33], while we use a constant non-blank reward ρ to shift the logits produced by the LM for RNN-T [34] networks [35, 36].”
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition (2015, methodology): “CTC has also been applied to end-to-end training with phoneme targets and n-gram language models using FSTs in [26, 27, 6].”
State-of-the-Art Speech Recognition with Sequence-to-Sequence Models (2017, background): “Similar to [27, 28], at the current frame, t, these features are stacked with 3 frames to the left and downsampled to a 30ms frame rate.”
A Comparison of Modeling Units in Sequence-to-Sequence Speech Recognition with the Transformer on Mandarin Chinese (2018, background): “Similar to [17, 18], at the current frame t, these features are stacked with 3 frames to the left and downsampled to a 30ms frame rate.”
Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinese (2018, background): “Similar to [19, 20], at the current frame t, these features are stacked with 3 frames to the left and downsampled to a 30ms frame rate.”
Achieving Human Parity in Conversational Speech Recognition (2016, methodology): “Surprisingly, this is the case for both acoustic modeling [28, 29, 30, 31, 32, 33] and language modeling [34, 35, 36, 37].”
Self-Normalizing Neural Networks (2017, background): “Recurrent neural networks (RNNs) [15] achieved new levels at speech and natural language processing, for example at the TIMIT benchmark [10] or at language translation [29], and are already employed in mobile devices [24].”
The microsoft 2016 conversational speech recognition system (2016, methodology): “Surprisingly, this is the case for both acoustic modeling [8, 9, 10, 11] and language modeling [12, 13].”
Exploring architectures, data and units for streaming end-to-end speech recognition with RNN-transducer (2017, methodology): “To achieve noise robustness acoustic training data is distorted as described in [23].”
Improved training of end-to-end attention models for speech recognition (2018, background): “There was certain work to remove the GMM dependency in the pretraining [5], or to be able to train without an existing alignment [6–8], or to avoid the lexicon [9], which simpliﬁes the pretraining procedure but still is not end-to-end.”
An Analysis of Incorporating an External Language Model into a Sequence-to-Sequence Model (2017, neutral): “Similar to [23, 24], at the current frame, t, these features are stacked with 3 frames to the left and downsampled to a 30ms frame rate.”
Bytes Are All You Need: End-to-end Multilingual Speech Recognition and Synthesis with Bytes (2018, background): “Similar to [29, 30], at each current frame, these features are stacked with 3 consecutive frames to the left and then down-sampled to a 30ms frame rate.”
Towards Practical Verification of Machine Learning: The Case of Computer Vision Systems (2017, background): “, 299 × 299 × 3 images for Inception-V3 [46], 640×480 images for Google Cloud Vision API [2], or audio speech input using only one out of three original frames for speech recognition [43]).”
The Microsoft 2017 Conversational Speech Recognition System (2017, methodology): “For our LSTM-based acoustic models we use a bidirectional architecture (BLSTM) [34] without frame-skipping [11].”
From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognition (2019, background): “, “the cat”) [20, 36, 37], or a mixture of words and graphemes [38, 39].”
An Online Sequence-to-Sequence Model Using Partial Conditioning (2015, background): “or a Recurrent Network (Hinton et al., 2012; Abdel-Hamid et al., 2012; Graves et al., 2013b). More recently, a CTC approach was also proposed for this task (Graves et al., 2013a; Hannun et al., 2014; Sak et al., 2015). An important aspect of these approaches is that the model makes predictions at ev”