Feature engineering in Context-Dependent Deep Neural Networks for conversational speech transcription

By Frank Torsten Bernd Seide, Gang Li, Xie Chen, Dong Yu. 2011

694 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dong Yu
2. Jinyu Li
3. Li Deng
4. Yoshua Bengio
5. James R. Glass
6. Shinji Watanabe
7. Tara N. Sainath
8. Quoc V. Le

In Natural Language Processing

1. Li Deng
2. Dong Yu
3. Yoshua Bengio
4. Kyunghyun Cho
5. Jianfeng Gao
6. Jinyu Li
7. Quoc V. Le
8. Mohit Bansal

In Speech & Audio

1. Jinyu Li
2. Dong Yu
3. James R. Glass
4. Tara N. Sainath
5. Shinji Watanabe
6. Li Deng
7. Wei-Ning Hsu
8. Yu Zhang

In Signal Processing

1. Jinyu Li
2. Dong Yu
3. Li Deng
4. Shinji Watanabe
5. Tara N. Sainath
6. Yoshua Bengio
7. James R. Glass
8. Yifan Gong

Citations in the corpus

• Robust Speech Recognition via Large-Scale Weak Supervision (2022, methodology): “Further research demonstrated that the benefit of deep learning approaches to speech recognition increased with dataset size, improving from being only competitive with prior GMM-HMM systems when using just 3 hours of TIMIT training data for phone recognition to achieving a 30% word error rate reduc”
• Representation Learning: A Review and New Perspectives (2012, background): “Another variant (Seide et al., 2011b) pre-trains in a supervised way all the previously added layers at each step of the iteration, and in their experiments this discriminant variant yielded better results than unsupervised pre-training.”
• Deep Speech: Scaling up end-to-end speech recognition (2014, neutral): “. (GMM-HMM BMMI) [44] 18.6 33.0 25.8 Vesely et al. (DNN-HMM sMBR) [44] 12.6 24.1 18.4 Maas et al. (DNN-HMM SWB) [28] 14.6 26.3 20.5 Maas et al. (DNN-HMM FSH) [28] 16.0 23.7 19.9 Seide et al. (CD-DNN) [39] 16.1 n/a n/a Kingsbury et al. (DNN-HMM sMBR HF) [22] 13.3 n/a n/a Sainath et al. (CNN-HMM) [36]”
• Deep Learning: Methods and Applications (2014, methodology): “What was not obvious, however, is how to efficiently and effectively initialize DNN parameters and how the use of very large amounts of training data can alleviate the learning problem until more recently (Hinton et al. 2006; Hinton and Salakhutdinov, 2006; Bengio, 2009; Vincent et al., 2010; Deng e”
• Parallel training of Deep Neural Networks with Natural Gradient and Parameter Averaging (2014, methodology): “…in [19], because while it is well established that it improves results for small datasets, our understanding is that as the amount of training data gets larger, it eventually gives no improvement compared to a suitable random initialization or discriminative layer-wise backpropagation as in [20].”
• EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding (2015, background): “In recent years, the performance of ASR has been improved dramatically by the introduction of deep neural networks (DNNs) as acoustic models [1, 2, 3].”
• Deep Learning of Representations: Looking Forward (2013, methodology): “. As a side-effect of this training, a new representation is produced, which can be used as input for deeper layers. With the less common supervised pre-training (Bengio et al., 2007; Yu et al., 2010; Seide et al., 2011b), each additional layer is trained with a supervised objective (as part of a one”
• Highway long short-term memory RNNS for distant speech recognition (2015, methodology): “All models are randomly initialized without either generative or discriminative pretraining [24].”
• Feature Learning in Deep Neural Networks - A Study on Speech Recognition Tasks (2013, background): “However, these gains were not realized, as shown in Table 4 [9].”
• Speaker adaptive training in deep neural networks using speaker dependent bottleneck features (2016, background): “These layers can be positioned either towards the front [8, 9], middle [9] or towards the output layer [10].”
• Feature space maximum a posteriori linear regression for adaptation of deep neural networks (2014, methodology): “Transformation based methods have also been employed in [29], and adaptation is only performed on the transform network in order to reduce the number of parameters to be optimized.”
• Multiple-source adaptation theory and algorithms (2020, background): “…50, 60]; in natural language processing, different contexts result in different word choices and expression styles [4, 12, 22, 33, 38, 46, 54]; in speech recognition, speech varies by speakers and by domains, such as casual conversations, broadcast news, lectures [13, 37, 52], just to name a few.”
• Simple Recurrent Units for Highly Parallelizable Recurrence (2017, methodology): “ainath et al., 2015), all weights are randomly initialized from the uniform distribution with range [ 0:05;0:05], and all biases are initialized to 0 without generative or discriminative pretraining (Seide et al., 2011). All neural network models, unless explicitly stated otherwise, are trained with”
• A comprehensive study of deep bidirectional LSTM RNNS for acoustic modeling in speech recognition (2016, methodology): “We investigated the same pretraining scheme as we do for our FFNN where we start with one layer and add a layer after each epoch right before the output layer [45].”
• Conditional Teacher-student Learning (2019, methodology): “Many techniques have been proposed for speaker adaptation of deep acoustic models, such as regularization-based [25, 26, 27], transformation-based [28, 29], singular value decomposition-based”
• Deep neural network features and semi-supervised training for low resource speech recognition (2013, methodology): “The parameters of a DNN are often initialized using a pre-training algorithm before the network is trained to completion using error back-propagation [13, 14].”