Investigation of full-sequence training of deep belief networks for speech recognition

By Abdel-rahman Mohamed, Dong Yu, Li Deng. 2010

220 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Li Deng
2. Dong Yu
3. Jinyu Li
4. Yoshua Bengio
5. Jianfeng Gao
6. Xiaodong He
7. Quoc V. Le
8. Mohit Bansal

In Signal Processing

1. Dong Yu
2. Li Deng
3. Jinyu Li
4. Yifan Gong
5. Tara N. Sainath
6. M. Seltzer
7. Abdel-rahman Mohamed
8. Yoshua Bengio

Citations in the corpus

• Deep Learning: Methods and Applications (2014, methodology): “In implementing the above full-sequence learning algorithm for the DNN system as described in [260], the DNN weights are initialized using the frame-level cross entropy as the objective.”
• C3D: Generic Features for Video Analysis (2014, background): “…availability of powerful parallel machines (GPUs, CPU clusters), together with large amounts of training data, convolutional neural networks [33] have come back providing breakthroughs on many AI problems, re-lated to text [6, 34], speech recognition [35, 36] and image based problems [15, 30].”
• Parallel training of Deep Neural Networks with Natural Gradient and Parameter Averaging (2014, background): “Sequence training [22] is a term that has the the same meaning for DNNs that “discriminative training” [23] has in the speech recognition community for GMMs.”
• Reshaping deep neural network for fast decoding by node-pruning (2014, methodology): “The recently proposed Context-Dependent Deep-NeuralNetwork HMM (CD-DNN-HMM) has achieved significant improvement over the state-of-art CD-GMM-HMM for both phoneme recognition[1, 2] and large vocabulary continuous speech recognition(LVCSR) [3] tasks.”
• An Overview of Deep-Structured Learning for Information Processing (2011, methodology): “Various versions of deep-structured CRFs are usefully applied to phone recognition (Yu and Deng, 2010), spoken language identification (Yu, Wang, Karam, and Deng, 2010), and natural language processing (Yu, Wang, and Deng, 2010).”
• Improved Bottleneck Features Using Pretrained Deep Neural Networks (2011, methodology): “…the DNN-HMM which exploits the discriminative learning ability of pretrained DNNs and the sequential modeling ability of hidden Markov models (HMMs) outperformed the conventional Gaussian mixture model (GMM)-HMM for both phoneme recognition [6][7] and large vocabulary speech recognition [8] tasks.”
• Deep Convex Net: A Scalable Architecture for Speech Pattern Classification (2011, methodology): “We ran the DBN system of [7] on the same TIMIT data and found the corresponding frame-level phone state error rate be to 45.04% (which gave 22% phonetic recognition error rate after running a decoder with a standard bi-gram phonetic “language” model as reported in [7]).”
• Large Vocabulary Speech Recognition Using Deep Tensor Neural Networks (2012, methodology): “CD-DNN-HMM is a special artificial neural network (ANN) HMM hybrid system [7][8][9] proposed two decades ago for speech recognition.”
• Boosting attribute and phone estimation accuracies with deep neural networks for detection-based speech recognition (2012, methodology): “We extend the conventional shallow MLPs used in [12] to deep neural networks (DNNs) [13], which has been shown to have very good theoretical properties [14] and demonstrated superior performances for both phone [16][ 17 ] and word recognition [15][13][18][19].”
• Time series prediction of stock price using deep belief networks with intrinsic plasticity (2017, methodology): “It have already been applied successfully to solve many problems, such as classification[12][13][14], voice activity detection[15][16][17], information retrieval[18], traffic flow prediction[19], and forecasting exchange rate[20].”
• Recent advances in deep learning for speech research at Microsoft (2013, background): “…researchers at University of Toronto and other organizations, we at Microsoft have developed deep learning technology that has successfully replaced Gaussian mixtures for speech recognition and feature coding at an increasingly larger scale (e.g., [24][19][53][39][7][8][44][54][13][56][30][48]).”
• Revisiting Recurrent Neural Networks for robust ASR (2012, methodology): “In a Tandem system, a base feature such as a mel-scale cepstral coefficients (MFCCs) or perceptual linear prediction (PLP) is used as the input to an MLP trained on relevant labels (typically phones, or monophone HMM states).”
• Deep Learning and Its Applications to Signal and Information Processing [Exploratory DSP] (2011, background): “As such, deep networks that are learned with unsupervised DBN pretraining followed by the backpropagation fine-tuning are also called DBNs in the literature (e.g., [8] and [9]).”