Speech recognition with deep recurrent neural networks
By Alex Graves, Abdel-rahman Mohamed, Geoffrey E. Hinton. 2013
9,035 citations recorded by the source corpus.
Recurrent neural networks (RNNs) are a powerful model for sequential data. End-to-end training methods such as Connectionist Temporal Classification make it possible to train RNNs for sequence labelling problems where the input-output alignment is unknown. The combination of these methods with the Long Short-term Memory RNN architecture has proved particularly fruitful, delivering state-of-the-art results in cursive handwriting recognition. However RNN performance in speech recognition has so far been disappointing, with better results returned by deep feedforward networks. This paper investigates deep recurrent neural networks, which combine the multiple levels of representation that have proved so effective in deep networks with the flexible use of long range context that empowers RNNs. When trained end-to-end with suitable regularisation, we find that deep Long Short-term Memory RNNs achieve a test set error of 17.7% on the TIMIT phoneme recognition benchmark, which to our knowledge is the best recorded score.
Subfields: Artificial Intelligence, Natural Language Processing, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Quoc V. Le
- Jianfeng Gao
- Percy Liang
- Sergey Levine
- Kyunghyun Cho
- Ruslan Salakhutdinov
In Natural Language Processing
- Mohit Bansal
- Jianfeng Gao
- Graham Neubig
- Kyunghyun Cho
- Yoshua Bengio
- Richard Socher
- Percy Liang
- Luke S. Zettlemoyer
In Signal Processing
Citations in the corpus
- Adam: A Method for Stochastic Optimization (2014, methodology): “SGD proved itself as an efficient and effective optimization method that was central in many machine learning success stories, such as recent advances in deep learning (Deng et al., 2013; Krizhevsky et al., 2012; Hinton & Salakhutdinov, 2006; Hinton et al., 2012a; Graves et al., 2013).”
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (2014, background): “Those tasks include, but are not limited to, speech recognition [see, e.g., Graves et al., 2013] and machine translation [see, e.g., Sutskever et al., 2014, Bahdanau et al., 2014].”
- Generating Sequences With Recurrent Neural Networks (2013, background): “Long Short-term Memory (LSTM) [16] is an RNN architecture designed to be better at storing and accessing information than standard RNNs. LSTM has recently given state-of-the-art results in a variety of sequence processing tasks, including speech and handwriting recognition [10, 12].”
- Playing Atari with Deep Reinforcement Learning (2013, background): “Recent advances in deep learning have made it possible to extract high-level features from raw sensory data, leading to breakthroughs in computer vision [11, 22, 16] and speech recognition [6, 7].”
- Adaptive Computation Time for Recurrent Neural Networks (2016, positive): “The evidence that increased depth leads to more performant networks is by now inarguable [5, 4, 19, 9], and recent results show that increased sequence length can be similarly beneficial [31, 33, 25].”
- Large Memory Layers with Product Keys (2019, background): “code for reproducibility purposes.3 1 Introduction Neural networks are commonly employed to address many complex tasks such as machine translation [43], image classification [27] or speech recognition [16]. As more and more data becomes available for training, these networks are increasingly larger [”
- Neural Turing Machines (2014, background): “…they natively handle variable-length structures, they have recently been used in a variety of cognitive problems, including speech recognition (Graves et al., 2013; Graves and Jaitly, 2014), text generation (Sutskever et al., 2011), handwriting generation (Graves, 2013) and machine…”
- The Power of Scale for Parameter-Efficient Prompt Tuning (2021, background): “(2021) represent the prompts as the output of a BiLSTM (Graves et al., 2013), while we represent our prompts with a simple embedding table.”
- Unbounded cache model for online language modeling with open vocabulary (2017, background): “We refer the reader to [23] for details on this architecture.”
- Improving Neural Language Models with a Continuous Cache (2016, background): “The LSTM architecture is particularly interesting in the context of language modelling (Jozefowicz et al., 2016) and we refer the reader to Graves et al. (2013) for details on this architecture.”
- mixup: Beyond Empirical Risk Minimization (2017, background): “Data augmentation lies at the heart of all successful applications of deep learning, ranging from image classification (Krizhevsky et al., 2012) to speech recognition (Graves et al., 2013; Amodei et al., 2016).”
- Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, background): “, 2006; Graves & Jaitly, 2014), RNN-T (Graves, 2012; Graves et al., 2013) and seq2seq (Chan et al.”
- DeepSetNet: Predicting Sets with Deep Neural Networks (2016, background): “A sudden success in multiple applications including voice recognition [17], machine translation [40] and image classification [22], has sparked the deployment of deep learning methods throughout numerous research areas.”
- Perceiver: General Perception with Iterative Attention (2021, methodology): “If we move to audio, then the merits of a 2D grid are no longer as clear, and a different type of model, such as 1D convolutions or an LSTM (Hochreiter & Schmidhuber, 1997; Graves et al., 2013), may be warranted instead.”
- Adversarial Feature Learning (2016, background): “In other perceptual domains such as natural language processing or speech recognition, deep networks have proven highly effective as well [2, 11, 27, 29].”
- Learning to Execute (2014, methodology): “Other interesting applications of recurrent neural networks includes speech recognition (Robinson et al., 1996; Graves et al., 2013), machine translation (Cho et al.”