Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition

By George E. Dahl, Dong Yu, Li Deng, A. Acero. 2012

3,091 citations recorded by the source corpus.

We propose a novel context-dependent (CD) model for large-vocabulary speech recognition (LVSR) that leverages recent advances in using deep belief networks for phone recognition. We describe a pre-trained deep neural network hidden Markov model (DNN-HMM) hybrid architecture that trains the DNN to produce a distribution over senones (tied triphone states) as its output. The deep belief network pre-training algorithm is a robust and often helpful way to initialize deep neural networks generatively that can aid in optimization and reduce generalization error. We illustrate the key components of our model, describe the procedure for applying CD-DNN-HMMs to LVSR, and analyze the effects of various modeling choices on performance. Experiments on a challenging business search dataset demonstrate that CD-DNN-HMMs can significantly outperform the conventional context-dependent Gaussian mixture model (GMM)-HMMs, with an absolute sentence accuracy improvement of 5.8% and 9.2% (or relative error reduction of 16.0% and 23.2%) over the CD-GMM-HMMs trained using the minimum phone error rate (MPE) and maximum-likelihood (ML) criteria, respectively.

Subfields: Artificial Intelligence, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yoshua Bengio
2. Dong Yu
3. Quoc V. Le
4. Jianfeng Gao
5. Li Deng
6. Mohit Bansal
7. Kyunghyun Cho
8. Jinyu Li

In Natural Language Processing

1. Jianfeng Gao
2. Kyunghyun Cho
3. Yoshua Bengio
4. Mohit Bansal
5. Graham Neubig
6. Quoc V. Le
7. Li Deng
8. Dong Yu

In Signal Processing

1. Dong Yu
2. Jinyu Li
3. Li Deng
4. Shinji Watanabe
5. Yoshua Bengio
6. Tara N. Sainath
7. James R. Glass
8. Yu Zhang

Citations in the corpus

• Can Active Memory Replace Attention? (2016, background): “Recent successes of deep neural networks have spanned many domains, from computer vision [1] to speech recognition [2] and many other tasks.”
• Sequence to Sequence Learning with Neural Networks (2014, background): “Deep Neural Networks (DNNs) are extremely powerful machine learning models that achieve excellent performance on difficult problems such as speech recognition [13, 7] and visual object recognition [19, 6, 21, 20].”
• Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014, background): “Deep neural networks have shown great success in various applications such as objection recognition (see, e.g., (Krizhevsky et al., 2012)) and speech recognition (see, e.g., (Dahl et al., 2012)).”
• Deeply-Supervised Nets (2014, background): “The application of deep learning (DL) techniques to large training sets has yielded significant performance gains in image classification [11, 15] and speech recognition [4].”
• Improving neural networks by preventing co-adaptation of feature detectors (2012, methodology): “Recently, deep, pre-trained, feedforward neural networks that map a short sequence of frames into a probability distribution over HMM states have been shown to outperform traditional Gaussian mixture models on both TIMIT [12] and a variety of more realistic large vocabulary tasks [6, 10].”
• Playing Atari with Deep Reinforcement Learning (2013, background): “Recent advances in deep learning have made it possible to extract high-level features from raw sensory data, leading to breakthroughs in computer vision [11, 22, 16] and speech recognition [6, 7].”
• Adaptive Computation Time for Recurrent Neural Networks (2016, positive): “The evidence that increased depth leads to more performant networks is by now inarguable [5, 4, 19, 9], and recent results show that increased sequence length can be similarly beneficial [31, 33, 25].”
• Unsupervised Pretraining for Sequence to Sequence Learning (2016, background): “Unsupervised pretraining has been intensively studied in the past years, most notably is the work by Dahl et al. (2012) who found that pretraining with deep belief networks improved feedforward acoustic models.”
• MS MARCO: A Human Generated MAchine Reading COmprehension Dataset (2016, background): “(2012); Dahl et al. (2012). As these types of assistants rise in popularity, consumers are finding it more convenient to ask a question and quickly get an answer through voice assistance as opposed to navigating through a search engine result page and web browser.”
• End-to-End Training of Deep Visuomotor Policies (2015, background): “, 2012) speech recognition (Dahl et al., 2012), and even playing video games (Mnih et al.”
• GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10, 000 Hours of Transcribed Audio (2021, background): “Various system architectures, from hybrid [1] to end-to-end [2], are proposed, and state-of-the-art results on standard benchmarks are being frequently updated.”
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition (2019, methodology): “Deep Learning has been applied successfully to Automatic Speech Recognition (ASR) [1], where the main focus of research has been designing better network architectures, for example, DNNs [2], CNNs [3], RNNs [4] and end-to-end models [5, 6, 7].”
• Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, methodology): “More recently DNNs have become a fixture in the ASR pipeline with almost all state of the art speech work containing some form of deep neural network (Mohamed et al., 2011; Hinton et al., 2012; Dahl et al., 2011; N. Jaitly & Vanhoucke, 2012; Seide et al., 2011).”
• Light Gated Recurrent Units for Speech Recognition (2018, background): “ReLU units have been shown to be more effective than sigmoid non-linearities for feed-forward DNNs [33], [34].”
• Massively Parallel Methods for Deep Reinforcement Learning (2015, methodology): “…methods have recently achieved state-of-the-art results in vision and speech domains (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Szegedy et al., 2014; Graves et al., 2013; Dahl et al., 2012), mainly due to their ability to automatically learn high-level features from a supervised signal.”