A comprehensive study of deep bidirectional LSTM RNNS for acoustic modeling in speech recognition

By Albert Zeyer, Patrick Doetsch, Paul Voigtlaender, Ralf Schlüter, Hermann Ney. 2016

175 citations recorded by the source corpus.

Recent experiments show that deep bidirectional long short-term memory (BLSTM) recurrent neural network acoustic models outperform feedforward neural networks for automatic speech recognition (ASR). However, their training requires a lot of tuning and experience. In this work, we provide a comprehensive overview over various BLSTM training aspects and their interplay within ASR, which has been missing so far in the literature. We investigate on different variants of optimization methods, batching, truncated backpropagation, and regularization techniques such as dropout, and we study the effect of size and depth, training models of up to 10 layers. This includes a comparison of computation times vs. recognition performance. Furthermore, we introduce a pretraining scheme for LSTMs with layer-wise construction of the network showing good improvements especially for deep networks. The experimental analysis mainly was performed on the Quaero task, with additional results on Switchboard. The best BLSTM model gave a relative improvement in word error rate of over 15% compared to our best feed-forward baseline on our Quaero 50h task. All experiments were done using RETURNN and RASR, RWTH's extensible training framework for universal recurrent neural networks and ASR toolkit. The training configuration files are publicly available.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

RWTH ASR Systems for LibriSpeech: Hybrid vs Attention - w/o Data Augmentation (2019, methodology): “Newbob learning rate scheduling [13] is applied to control the learning rate reduction with a learning rate decay rate of 0.”
Improved training of end-to-end attention models for speech recognition (2018, methodology): “Conventional speech recognition systems [1] with neural network (NN) based acoustic models using the hybrid hidden Markov models (HMM) / NN approach [2, 3] usually operate on the phone level, given a phonetic pronunciation lexicon (from phones to words).”
Language Modeling with Deep Transformers (2019, methodology): “Our baseline acoustic model is based on multi-layer bi-directional LSTM [37].”
A Comparison of Transformer and LSTM Encoder Decoder Models for ASR (2019, background): “It also tries to reduce any assumptions being made about the data, and thus better performance can potentially be expected compared with the conventional system, the hybrid hidden Markov model (HMM) - neural network (NN) approach on phoneme level [9–12].”
A Hitting Time Analysis of Stochastic Gradient Langevin Dynamics (2017, background): “Theoretical understanding of the value of gradient noise is incomplete.”
High-performance brain-to-text communication via handwriting (2021, methodology): “Our methods are similar to neural network methods used in automatic speech recognition (Hinton et al., 2012; Graves et al., 2013; Zeyer et al., 2017; Xiong et al., 2017), but with some key changes made to achieve high performance on neural activity in a highly data-limited regime (1-10 hours of data”
End-to-End Speech Recognition: A Survey (2023, background): “Supervisedlayer-wisepretraininghas beensuccessfullyappliedforclassical[5],[145],aswellas attention-basedASRmodels[146],whichcanbecombinedwith intermediatesub-samplingschemes[147],andmodelgrow-ing[148].”
The Rwth Asr System for Ted-Lium Release 2: Improving Hybrid Hmm With Specaugment (2020, methodology): “Greedy layer-wise pre-training [19] and Newbob learning rate scheduling [20] with a decay factor of 0.9 are applied.”
A comprehensive analysis on attention models (2019, background): “momentum (Nadam) [Dozat, 2015], Newbob learning rate scheduling [Zeyer et al., 2017], and focal loss [Lin et al.”
Returnn: The RWTH extensible training framework for universal recurrent neural networks (2016, background): “Recurrent neural networks (RNNs) and in particular LSTMs [1] now dominate most sequential learning tasks including automatic speech recognition (ASR) [2, 3], statistical machine translation (SMT) [4], and image caption generation [5].”