Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling

By Brian Kingsbury. 2009

305 citations recorded by the source corpus.

Acoustic models used in hidden Markov model/neural-network (HMM/NN) speech recognition systems are usually trained with a frame-based cross-entropy error criterion. In contrast, Gaussian mixture HMM systems are discriminatively trained using sequence-based criteria, such as minimum phone error or maximum mutual information, that are more directly related to speech recognition accuracy. This paper demonstrates that neural-network acoustic models can be trained with sequence classification criteria using exactly the same lattice-based methods that have been developed for Gaussian mixture HMMs, and that using a sequence classification criterion in training leads to considerably better performance. A neural network acoustic model with 153K weights trained on 50 hours of broadcast news has a word error rate of 34.0% on the rt04 English broadcast news test set. When this model is trained with the state-level minimum Bayes risk criterion, the rt04 word error rate is 27.7%.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Deep Learning: Methods and Applications (2014, methodology): “The solution is provided by replacing the frame-based, cross-entropy training criterion with the fullsequence-based maximum mutual information optimization objective, in a similar way to defining the training objective for the shallow neural network interfaced with an HMM [194].”
End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results (2014, methodology): “Once a per-frame classiﬁer is trained, the whole system including the acoustic model and the HMM can be tuned jointly on full sentences to minimize the decoding error (LeCun et al. , 1998; He et al. , 2008; Kingsbury, 2009; Vesel et al. , 2013).”
Fast and accurate recurrent neural network acoustic models for speech recognition (2015, methodology): “A number of sequence-level discriminative training criteria incorporating the lexical and language model constraints used in speech decoding have been shown to improve the performance of DNN and RNN acoustic models bootstrapped with CE [19, 20, 12, 21, 4] or CTC training criteria [5].”
Sequence-to-Sequence Learning as Beam-Search Optimization (2016, positive): “The necessity of pre-training in this instance is consistent with the findings of other authors who train non-local neural models (Kingsbury, 2009; Sak et al., 2014; Andor et al., 2016; Ranzato et al., 2016).”
A tutorial survey of architectures, algorithms, and applications for deep learning (2014, methodology): “A closely related full-sequence training method is carried out with success for a shallow neural network [140] and for a deep one [141].”
Enhancing the TED-LIUM Corpus with Selected Data for Language Modeling and More TED Talks (2014, methodology): “…align these closed captions to the acoustic data to produce proper transcriptions, we ﬁrst built a deep neural network (DNN) based on state-level minimum Bayes risk (sMBR) (Kingsbury, 2009) discriminative criterion upon fMMI baseline acoustic models similar to the ones described in section 3.3..”
Recurrent deep neural networks for robust speech recognition (2014, methodology): “The loss functions based on discriminative training criteria such as sMBR [26], MMI and MPE/MWE [27] have also been used for ASR.”
On the compression of recurrent neural networks with an application to LVCSR acoustic modeling for embedded speech recognition (2016, methodology): “Systems are additionally adapted using the sMBR criterion [26, 27] on a set of”
On parallelizability of stochastic gradient descent for speech DNNS (2014, methodology): “A CD-DNN-HMM models the posterior probability P (s|o) of a tied triphone state, or senone s [16, 1], given an observation vector o.”
Exemplar-Based Sparse Representation Features: From TIMIT to LVCSR (2011, methodology): “The acoustic model is trained on 50 hours of data from the 1996 and 1997 English Broadcast News Speech Corpora.”
Advances in Very Deep Convolutional Neural Networks for LVCSR (2016, unknown): “Sequence training was introduced to neural network training in [28].”
Integrating Deep Neural Networks into Structural Classification Approach based on Weighted Finite-State Transducers (2012, methodology): “The dMMI objective function is closely related to the sequence-level minimum Bayes risk training of neural networks [15].”
Hybrid Autoregressive Transducer (HAT) (2020, background): “some extent by introducing discriminative approaches for acoustic modeling [7, 8, 9] and language modeling [10].”
Learning filter banks within a deep neural network framework (2013, methodology): “Our filter bank learning experiments are performed on a 50-hr English Broadcast News (BN) task [12].”
Knowledge distillation for small-footprint highway networks (2016, background): “,xT } and Y as the sequence of labels, where T is the length of the signal, the loss function from the state-level minimum Bayesian risk criterion(sMBR) [19, 20] is defined as”
Investigation of full-sequence training of deep belief networks for speech recognition (2010, background): “Using neural networks to discriminate between sequences has indeed been proposed in the past (e.g., [4][ 10 ][11][16], but it was done in shallow architectures and not in an integrative manner.”
Collecting Resources in Sub-Saharan African Languages for Automatic Speech Recognition: a Case Study of Wolof (2016, methodology): “To build DNNs, we trained the network using state-level minimum Bayes risk (Kingsbury, 2009) (sMBR) and the network had seven layers, each of the six hidden layers had 1024 hidden units.”
Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model (2019, methodology): “They are trained using connectionist temporal classi-ﬁcation (CTC) [33], followed by state-level minimum Bayes risk [34, 35].”