Sequence-discriminative training of deep neural networks

By Karel Veselý, Arnab Ghoshal, Lukáš Burget, Daniel Povey. 2013

765 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Model Architectures, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Optimization & Training

In Signal Processing

Citations in the corpus

Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, background): “pe of convolution was ﬁrst proposed for neural networks in speech more than 25 years ago [67]. Many neural network speech models have a ﬁrst layer that processes input frames with some context window [16, 66]. This can be viewed as a temporal convolution with a stride of one. Additionally, sub-sampl”
Light Gated Recurrent Units for Speech Recognition (2018, methodology): “he effectiveness of this architectural variation. Note also that the GRU systems signiﬁcantly outperform the DNN baseline, even when the latter is based on sequence discriminative training (DNN+sMBR) [67]. Tab. XI splits the ASR performance of the real test set into the four noisy categories. Li-GRU”
Deep Speech: Scaling up end-to-end speech recognition (2014, methodology): “The model from Vesely et al. (DNN-GMM sMBR) [44] uses a sequence based loss function on top of a DNN after using a typical hybrid DNN-HMM system to realign the training set.”
Achieving Human Parity in Conversational Speech Recognition (2016, background): “As noted in [52, 53], the necessary gradient for use in backpropagation is a simple function of the posterior probability of a particular acoustic model state at a given time, as computed by summing over all possible word sequences in an unconstrained manner.”
Parallel training of Deep Neural Networks with Natural Gradient and Parameter Averaging (2014, neutral): “The high-level details of our lattice-based training procedure are similar to [29], but note that in that paper we describe an alternative implementation of deep neural nets (the “nnet1” setup) that exists within Kaldi; this paper is about the alternative “nnet2” setup.”
End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results (2014, methodology): “Once a per-frame classifier is trained, the whole system including the acoustic model and the HMM can be tuned jointly on full sentences to minimize the decoding error (LeCun et al., 1998; He et al., 2008; Kingsbury, 2009; Vesel et al., 2013).”
Fast and accurate recurrent neural network acoustic models for speech recognition (2015, methodology): “A number of sequence-level discriminative training criteria incorporating the lexical and language model constraints used in speech decoding have been shown to improve the performance of DNN and RNN acoustic models bootstrapped with CE [19, 20, 12, 21, 4] or CTC training criteria [5].”
The microsoft 2016 conversational speech recognition system (2016, background): “As noted in [32, 33], the necessary gradient for use in backpropagation is a simple function of the posterior probability of a particular acoustic model state at a given time, as computed by summing over all possible word sequences in an unconstrained manner.”
Semi-Supervised Speech Recognition via Local Prior Matching (2020, background): “Luckily, the denominator in equation 2 can often be approximated by summing over a set of hypotheses proposed by a trained ASR model, as done in sequence discriminative training (Povey, 2005; Veselỳ et al., 2013) and differentiable beam search decoding (Collobert et al., 2019).”
Experiments on Parallel Training of Deep Neural Network using Model Averaging (2015, methodology): “In this work, we report speech recognition results on the 300 hour Switchboard conversational telephone speech task (Switchboard-1 Release 2).”
Transformer-Based Acoustic Modeling for Hybrid Speech Recognition (2019, methodology): “We focus on cross-entropy (CE) trained models and only selectively perform sMBR [35] training on top of the best CE setup.”
The DIRHA-ENGLISH corpus and related tasks for distant-speech recognition in domestic environments (2015, methodology): “Finally, the DNN baseline ( DNN ), trained with the Karel’s recipe [32], is composed of 6 hidden layers of 1024 neurons, with a context window of 11 consecutive frames (5 before and 5 after the current frame) and an initial learning rate of 0.008.”
The NTT CHiME-3 system: Advances in speech enhancement and recognition for mobile multi-microphone devices (2015, methodology): “This baseline represents today’s standard, utilising sequence-trained deep neural network (DNN) acoustic models [6].”
Rectifier Nonlinearities Improve Neural Network Acoustic Models (2013, methodology): “tions during DNN training, and claim state-of-the-art results on many challenging tasks in speech recognition (Hinton et al., 2012; Kingsbury et al., 2012; Vesely et al., 2013).”