Purely Sequence-Trained Neural Networks for ASR Based on Lattice-Free MMI

By Daniel Povey, Vijayaditya Peddinti, Daniel Galvez, Pegah Ghahremani, Vimal Manohar, Xingyu Na, Yiming Wang, Sanjeev Khudanpur. 2016

891 citations recorded by the source corpus.

In this paper we describe a method to perform sequence-discriminative training of neural network acoustic models without the need for frame-level cross-entropy pre-training. We use the lattice-free version of the maximum mutual information (MMI) criterion: LF-MMI. To make its computation feasible we use a phone n-gram language model, in place of the word language model. To further reduce its space and time complexity we compute the objective function using neural network outputs at one third the standard frame rate. These changes enable us to perform the computation for the forward-backward algorithm on GPUs. Further the reduced output frame-rate also provides a signiﬁcant speed-up during decoding. We present results on 5 different LVCSR tasks with training data ranging from 100 to 2100 hours. Models trained with LF-MMI provide a relative word error rate reduction of ∼ 11 . 5% , over those trained with cross-entropy objective function, and ∼ 8% , over those trained with cross-entropy and sMBR objec-tive functions. A further reduction of ∼ 2 . 5% , relative, can be obtained by ﬁne tuning these models with the word-lattice based sMBR objective function.

Subfields: Artificial Intelligence, Control and Systems Engineering, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Speech & Audio

In Signal Processing

Citations in the corpus

GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10, 000 Hours of Transcribed Audio (2021, background): “To take the English speech recognition task as an example, the Wall Street Journal corpus, which consists of 80 hours of narrated news articles [3], is almost 20 years old, and has a word error rate (WER) of 2.32% on its eval92 benchmark [4].”
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition (2019, neutral): “;c) = (0.3, 0.05), (0.2, 0.0125) and (0.1, 0.025) respectively. Table 5: Switchboard 300h WERs (%). Method No LM With LM SWBD CH SWBD CH HMM Vesel´y et al., (2013) [41] 12.9 24.5 Povey et al., (2016) [30] 9.6 19.3 Hadian et al., (2018) [42] 9.3 18.9 Zeyer et al., (2018) [24] 8.3 17.3 CTC Zweig et al”
TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation (2018, methodology): “They were trained using LF-MMI criterion [9] and 3-fold reduced frame rate.”
Improving Transformer-based Speech Recognition Using Unsupervised Pre-training (2019, neutral): “TDNN-hybrid [20] 23.”
Wav2Letter++: A Fast Open-source Speech Recognition System (2018, methodology): “The Kaldi recipe for LF-MMI does not synchronize gradients for each SGD update; the perepoch time is still more than 20x slower.”
ESPnet: End-to-End Speech Processing Toolkit (2018, methodology): “Note that several benchmarks including HKUST and CSJ score comparable/superior performance to the state-of-the-art hybrid DNN/HMM systems based on lattice-free maximum mutual information training [27].”
Letter-Based Speech Recognition with Gated ConvNets (2017, methodology): “Recent state-of-the-art systems leverage the MMI criterion (Povey et al., 2016).”
The microsoft 2016 conversational speech recognition system (2016, methodology): “In [19], a language model on phonemes is used instead.”
The CAPIO 2017 Conversational Speech Recognition System (2017, methodology): “The cross entropy objective function was also applied as an extra regularization as well as leaky HMM to avoid overfitting [27].”
Improved training of end-to-end attention models for speech recognition (2018, background): “However, as the original input is continuous, some sort of downscaling in the time dimension works, such as concatenation in the feature dimension of consecutive time-frames [7,24,42,50].”
WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition (2021, methodology): “Kaldi Benchmark(5) The Kaldi baseline implements a classical chain model [29] using various amounts of WeNetSpeech data (i.”
Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces (2018, methodology): “The Raspberry Pi 3 model uses 7 layers, and is trained with the lattice-free Maximum Mutual Information (MMI) criterion [43], using natural gradient descent, with a learning rate of 0.”
Exploring neural transducers for end-to-end speech recognition (2017, neutral): “BLSTM + LF MMI [20] 8.”
Lattice-free State-level Minimum Bayes Risk Training of Acoustic Models (2018, methodology): “Note that the leaky hidden Markov model (leaky-HMM) technique, which adds transitions among all HMM-states, was also proposed for LF-MMI [8] to show its effectiveness as a regularizer.”
The Microsoft 2017 Conversational Speech Recognition System (2017, methodology): “Based on the approaches of [41] and [42], the denominator graph is a full trigram LM over phones and senones.”
End-to-End Speech Recognition: A Survey (2023, methodology): “…generative GMMs or neural networks to model the output distribution of acoustic frames; frame-level alignments to train neural network acoustic models may be obtained by force-alignment from a base GMM-HMM systems, but direct sequence training not requiring initial alignments is also possible [69].”
Fast Transformers with Clustered Attention (2020, methodology): “All transformers are trained with latticefree MMI loss [22] and as inputs we use 80-dimensional filter-bank features with fixed positional embeddings.”
Semi-Supervised Training of Acoustic Models Using Lattice-Free MMI (2018, methodology): “We assume we have a seed neural network model trained with LF-MMI on only the supervised data.”
Joint CTC/attention decoding for end-to-end speech recognition (2017, methodology): “2 Although the proposed method did not reach the performance obtained by a time delayed neural network (TDNN) with lattice-free sequence discriminative training (Povey et al., 2016), our recent work scored 28.”