Error back propagation for sequence training of Context-Dependent Deep NetworkS for conversational speech transcription

By Hang Su, Gang Li, Dong Yu, Frank Torsten Bernd Seide. 2013

153 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dong Yu
2. Jinyu Li
3. Li Deng
4. Yoshua Bengio
5. Tara N. Sainath
6. Jianfeng Gao
7. James R. Glass
8. Shinji Watanabe

In Signal Processing

1. Jinyu Li
2. Dong Yu
3. Li Deng
4. Tara N. Sainath
5. Yifan Gong
6. Shinji Watanabe
7. Yoshua Bengio
8. S. Renals

Citations in the corpus

• Deep Learning: Methods and Applications (2014, methodology): “While the batch-mode, second-order Hessian-free techniques prove successful for full-sequence training of large-scale DNN-HMM systems, the success of the first-order stochastic gradient descent methods is also reported recently (Su et al., 2013).”
• Fast and accurate recurrent neural network acoustic models for speech recognition (2015, methodology): “A number of sequence-level discriminative training criteria incorporating the lexical and language model constraints used in speech decoding have been shown to improve the performance of DNN and RNN acoustic models bootstrapped with CE [19, 20, 12, 21, 4] or CTC training criteria [5].”
• English Conversational Telephone Speech Recognition by Humans and Machines (2017, methodology): “The training consists of 14 passes of cross-entropy followed by 1 pass of SGD sequence training using the boosted MMI criterion [16] smoothed by adding the scaled gradient of the cross-entropy loss [17].”
• A scalable approach to using DNN-derived features in GMM-HMM based acoustic modeling for LVCSR (2013, methodology): “The result reported in [30] by further fine-tuning the DNN-HMM with a sequence based MMI criterion is also included in Table 5 for comparison.”
• Lattice-free State-level Minimum Bayes Risk Training of Acoustic Models (2018, methodology): ““Lattice-free” means that the denominator computation is done without conducting any lattice generation procedure, which is required when using the conventional MMI training [6, 7].”
• Using Recurrent Neural Networks for Slot Filling in Spoken Language Understanding (2015, methodology): “The R-CRF also differs from works in sequence training of DNN/HMM hybrid systems [46]–[48] for speech recognition, which use DNNs and HMMs, in that R-CRF uses the CRF objective and RNNs.”
• Advances in Very Deep Convolutional Neural Networks for LVCSR (2016, methodology): “• Regularization of ST by adding the gradient of crossentropy loss, as proposed in [13].”
• Differentiable Weighted Finite-State Transducers (2020, methodology): “In some cases, WFSTs are used statically to incorporate fixed lattices in discriminative sequence criteria (Kingsbury, 2009; Kingsbury et al., 2012; Su et al., 2013; Veselỳ et al., 2013).”
• Recent advances in deep learning for speech research at Microsoft (2013, background): “…researchers at University of Toronto and other organizations, we at Microsoft have developed deep learning technology that has successfully replaced Gaussian mixtures for speech recognition and feature coding at an increasingly larger scale (e.g., [24][19][53][39][7][8][44][54][13][56][30][48]).”
• On training the recurrent neural network encoder-decoder for large vocabulary end-to-end speech recognition (2016, background): “The past few years have seen significant advancements in speech recognition based on this hybrid architecture including using different neural network architectures [4, 5, 6], sequence training [7, 8, 9] and speaker adaptation [10, 11, 12].”
• Large scale deep neural network acoustic modeling with semi-supervised training data for YouTube video transcription (2013, background): “5% absolute which may be due to a missing heuristic [23] or limited gains from sequence training on a larger, semi-supervised training set.”
• The RWTH/UPB/FORTH System Combination for the 4th CHiME Challenge Evaluation (2016, methodology): “In order to prevent overfitting we used CE-smoothing [32] with a factor of 0 .”
• Building DNN acoustic models for large vocabulary speech recognition (2014, methodology): “Discriminative loss functions were initially developed for GMM acoustic models [38], [39], [40], [41], but were recently applied to DNN acoustic model training [4], [3], [42].”
• Sequence discriminative distributed training of long short-term memory recurrent neural networks (2014, methodology): “ASGD has also been used for sequence discriminative training of DNN models [17, 23].”
• Combining stochastic average gradient and Hessian-free optimization for sequence training of deep neural networks (2013, methodology): “This training is often carried out using a Hessian-free (HF) quasi-Newton approach, although other methods such as stochastic gradient descent (SGD) have also recently been applied successfully [3].”