Roles of Pre-Training and Fine-Tuning in Context-Dependent DBN-HMMs for Real-World Speech Recognition

By Dong Yu, Li Deng, George E. Dahl. 2010

224 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Li Deng
2. Dong Yu
3. Jinyu Li
4. Yoshua Bengio
5. Jianfeng Gao
6. Tara N. Sainath
7. Kyunghyun Cho
8. Quoc V. Le

Citations in the corpus

• Deep Learning: Methods and Applications (2014, methodology): “This success is a result of close academic-industrial collaboration, initiated at Microsoft Research, with the involved researchers identifying and acutely attending to the industrial need for large-scale deployment [68, 89, 109, 161, 323, 414].”
• Feature Learning in Deep Neural Networks - A Study on Speech Recognition Tasks (2013, methodology): “In all the following discussions, we use DNNs in the framework of the CD-DNN-HMM [6–10] and use speech recognition as our classification task.”
• Permutation invariant training of deep models for speaker-independent multi-talker speech separation (2016, background): “Motivated by the success of deep learning techniques in singletalker ASR [1, 2, 3, 4] , researchers have developed many deep learning techniques for speech separation in recent years.”
• Sequence classification using the high-level features extracted from deep neural networks (2014, unknown): “The use of fully-connected feed-forward deep neural networks (DNNs) in the architecture of DNN-HMM (hidden Markov Model) has dramatically reduced speech recognition errors in recent years [4][5][8][9][19][21][22][28][29][36][37].”
• Learning small-size DNN with output-distribution-based criteria (2014, methodology): “Context-dependent deep neural network hidden Markov model (CD-DNN-HMM) has been shown, by many groups [1][2][3][4][5][6], to outperform the conventional Gaussian mixture model (GMM)-HMMs on many automatic speech recognition (ASR) tasks.”
• Integrating Deep Neural Networks into Structural Classification Approach based on Weighted Finite-State Transducers (2012, methodology): “For example, [8] used the HMM states of clustered triphones as the output target classes of DNNs.”
• A Primal-Dual Method for Training Recurrent Neural Networks Constrained by the Echo-State Property (2013, background): “…been created in recent years using fully-connected deep neural networks (DNN) that drastically cut errors in large vocabulary speech recognition (Yu et al., 2010; Dahl et al., 2011; Seide et al., 2011; Dahl et al., 2012; Kingsbury et al., 2012; Hinton et al., 2012; Deng et al., 2013b,c; Yu et…”
• Optimization Techniques to Improve Training Speed of Deep Neural Networks for Large Speech Tasks (2013, background): “It was shown in [7], that DNN performance improves with increasing training data.”
• Large Vocabulary Speech Recognition Using Deep Tensor Neural Networks (2012, methodology): “Recently we proposed and developed the context-dependent deep neural network (DNN) hidden Markov models (HMMs) (CD-DNN-HMMs) for large vocabulary speech recognition [ 1 ][2][3][4][5].,CD-DNN-HMMs have achieved 16% [ 1 ][2] and 33% [3][4][5][6] relative recognition error reduction over strong, discri”
• Conversational Speech Transcription Using Context-Dependent Deep Neural Networks (2012, unknown): “Only recently did Yu et al. discover that doing so is not only feasible but works very well [ 5 ].,DNN-HMMs [ 5 , 6], apply the classical ANN-HMMs of the 90’s to traditional tied-state triphones directly, exploiting Hinton’s deep-belief-network (DBN) pre-training procedure.,This was shown to lead to”
• 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs (2014, methodology): “At present, the best context-dependent deep-neural-network HMMs, or CD-DNN-HMMs [1, 2], are trained primarily with error back-propagation, or BP.”
• Deep neural network features and semi-supervised training for low resource speech recognition (2013, background): “Networks trained from pre-trained weights are observed to be well regularized and converge to a better local optimum than randomly initialized networks [15, 16].”
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, background): “on the use of fully connected deep neural networks (DNNs) and deep autoencoders around 2010 [7]–[17].”