Discriminative Training for Large-Vocabulary Speech Recognition Using Minimum Classification Error

By Erik McDermott, Timothy J. Hazen, Jonathan Le Roux, Atsushi Nakamura, Shigeru Katagiri. 2007

638 citations recorded by the source corpus.

The minimum classification error (MCE) framework for discriminative training is a simple and general formalism for directly optimizing recognition accuracy in pattern recognition problems. The framework applies directly to the optimization of hidden Markov models (HMMs) used for speech recognition problems. However, few if any studies have reported results for the application of MCE training to large-vocabulary, continuous-speech recognition tasks. This article reports significant gains in recognition performance and model compactness as a result of discriminative training based on MCE training applied to HMMs, in the context of three challenging large-vocabulary (up to 100 k word) speech recognition tasks: the Corpus of Spontaneous Japanese lecture speech transcription task, a telephone-based name recognition task, and the MIT Jupiter telephone-based conversational weather information task. On these tasks, starting from maximum likelihood (ML) baselines, MCE training yielded relative reductions in word error ranging from 7% to 20%. Furthermore, this paper evaluates the use of different methods for optimizing the MCE criterion function, as well as the use of precomputed recognition lattices to speed up training. An overview of the MCE framework is given, with an emphasis on practical implementation issues

Subfields: Artificial Intelligence, Optimization & Training, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Speech & Audio

Citations in the corpus

Unsupervised Speech Recognition (2021, background): “Such models are often classified into two categories: hybrid (Juang et al., 1986; Young, 1996; Povey, 2005; Hinton et al., 2012; Abdel-Hamid et al., 2012; Bourlard and Morgan, 2012) or end-to-end (Graves et al., 2006; Graves, 2012; Chorowski et al., 2015; Rao et al., 2017).”
Semi-Supervised Speech Recognition via Local Prior Matching (2020, background): “Luckily, the denominator in equation 2 can often be approximated by summing over a set of hypotheses proposed by a trained ASR model, as done in sequence discriminative training (Povey, 2005; Veselỳ et al., 2013) and differentiable beam search decoding (Collobert et al., 2019).”
Unsupervised Adaptation With Discriminative Mapping Transforms (2009, background): “Here, only the final update formulae are given.”
Machine Learning Paradigms for Speech Recognition: An Overview (2013, methodology): “It was originated from MCE training of the generative model of Gaussian-mixture HMM [79], [102].”
Improving low-resource CD-DNN-HMM using dropout and multilingual DNN training (2013, methodology): “On top of the ML systems, discriminative training is performed using the boosted maximum mutual information (BMMI) criterion [21].”
Error back propagation for sequence training of Context-Dependent Deep NetworkS for conversational speech transcription (2013, methodology): “With 2000h of data, gains are 2–9% after realigned CE iterations.”
Hidden conditional random fields for phone classification (2005, background): “…resurgence of interest in discriminative methods for automatic speech recognition (ASR) due to the success of extended Baum-Welch (EBW) based techniques such as maximum mutual information (MMI) and minimum phone error (MPE) training in large vocabulary conversational speech recognition (LVCSR) [1].”
Large vocabulary continuous speech recognition with context-dependent DBN-HMMS (2011, neutral): “Index Terms— Speech recognition, deep belief network, context-dependent phone, LVCSR, DBN-HMM”
Statistical Parametric Speech Synthesis (2007, background): “It can also be viewed as estimating the following statistical model based on the ML criterion: (36) (37) (38) Equation (35) was iteratively minimized in an on-line fashion using the generalized probabilistic decent (GPD) algorithm (Katagiri et al., 1991), which has been used for minimum clas-si ﬁ ca”
Improving Robustness of Deep Neural Network Acoustic Models via Speech Separation and Joint Adaptive Training (2015, methodology): “The systems described in [38] and [44] are GMM-HMM systems, and use a feature enhancement frontend, along with discriminative features and discriminative acoustic modeling [32].”
Parameter Estimation of Statistical Models Using Convex Optimization (2010, methodology): “Moreover, a stochastic approximation method called generalized probabilistic descent (GPD) [18] was first proposed for MCE, and then an approximate second-order Quasi-Newton method called quickprop was applied to MCE and other criteria [26].”
Voice conversion in high-order eigen space using deep belief nets (2013, background): “It is often reported that such a discriminative approach performs better than a generative approach, such as GMM, in speech recognition and synthesis as well as in VC [18, 19].”
Lattice-free State-level Minimum Bayes Risk Training of Acoustic Models (2018, methodology): “The third technique is an MMI-based regularization that uses a combination of LF-sMBR and LF-MMI criteria similar to I-smoothing [20] or F-smoothing [7], as where λ is a scaling parameter.”
Unsupervised training and directed manual transcription for LVCSR (2010, methodology): “The MPE criterion can be optimised using the weaksense auxiliary function described in Povey (2003).”
Error Approximation and Minimum Phone Error Acoustic Model Estimation (2010, methodology): “These approximations assign an errorl(a) to each lattice arca such that the overall approximate error of each path is the sum of the errors associated with its composite arcs.”
Purely Sequence-Trained Neural Networks for ASR Based on Lattice-Free MMI (2016, methodology): “Traditionally this has been done by training a cross-entropy system, generating word lattices with a weak language model, and using these lattices as an approximation for all possible word sequences in the discriminative objective function– as was done when Gaussian Mixture Models were the state of ”
Context-dependent deep neural networks for commercial Mandarin speech recognition applications (2013, methodology): “Expectation-maximization (EM) algorithm is usually used for HMM training, while further recognition accuracy improvement can be achieved using discriminative training algorithms [1-3] such as MMI, MCE and MPE, etc.”