Finding consensus in speech recognition: word error minimization and other applications of confusion networks

By Lidia Mangu, Eric Brill, Andreas Stolcke. Computer Speech and Language, 2000

753 citations recorded by the source corpus.

We describe a new framework for distilling information from word lattices to improve the accuracy of the speech recognition output and obtain a more perspicuous representation of a set of alternative hypotheses. In the standard MAP decoding approach the recognizer outputs the string of words corresponding to the path with the highest posterior probability given the acoustics and a language model. However, even given optimal models, the MAP decoder does not necessarily minimize the commonly used performance metric, word error rate (WER). We describe a method for explicitly minimizing WER by extracting word hypotheses with the highest posterior probabilities from word lattices. We change the standard problem formulation by replacing global search over a large set of sentence hypotheses with local search over a small set of word candidates. In addition to improving the accuracy of the recognizer, our method produces a new representation of a set of candidate hypotheses that specifies the sequence of word-level confusions in a compact lattice format. We study the properties of confusion networks and examine their use for other tasks, such as lattice compression, word spotting, confidence annotation, and reevaluation of recognition hypotheses using higher-level knowledge sources.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/f7d08c4fdf359f5cf0946a4a86db52d273a59ba4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

The Dialog State Tracking Challenge Series: A Review (2016, background): “Speciﬁcally, the ASR may output an N-Best list of sentences, a word confusion network (Mangu et al., 2000), or a lattice; the SLU may output an N-Best list of interpretations.”
LIMSI’s Contribution to the WMT’16 Biomedical Translation Task (2016, background): “A CN is a weighted directed acyclic graph where all the paths go through all the nodes (Mangu et al., 2000).”
MVA Processing of Speech Features (2007, background): “Such combination can be done during feature extraction [9], lattice re-scoring [23], or output generation [24], [ 25 ].”
Confidence Estimation for Machine Translation (2004, methodology): “Typically these are not the same scores used as search criteria to establish the output word sequence in the first place, but rather posterior probabilities p(y|x) calculated from an nbest list or from a word lattice as in [20, 43].”
System Combination for Multi-document Summarization (2015, background): “System combination has enjoyed great success in many domains, such as automatic speech recognition (Fiscus, 1997; Mangu et al., 2000), machine translation (Frederking and Nirenburg,”
From Feedforward to Recurrent LSTM Neural Networks for Language Modeling (2015, methodology): “Additional improvements were obtained by applying confusion network rescoring (cf. [27]) on the lattices.”
Large Margin Training for Attention Based End-to-End Speech Recognition (2019, methodology): “Although n-best list or confusion network based methods can improve the efficiency [21, 22, 23], beam search decoding based on MAP is still one of the most commonly used evaluation methods in practice.”
Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces (2018, methodology): “rrectly decoded. Conﬁdence scoring is a notoriously hard problem in speech recognition [20, 59]. Our approach is based on the so-called “confusion network” representation of the hypotheses of the ASR [28, 57] (see Figure 5). A confusion network is a graph encoding, for each speech segment in an utte”
SRILM - an extensible language modeling toolkit (2002, methodology): “nbest-lattice — a tool to perform word error minimization on N-best lists [18] or construct confusion networks (“sausages”) [19].”
The RWTH/UPB/FORTH System Combination for the 4th CHiME Challenge Evaluation (2016, methodology): “The lattices are ﬁrst converted to individual confusion networks [27, 28] and the combination is performed by aligning the confusion networks in the order of increasing word error rate.”
The 2016 BBN Georgian telephone speech keyword spotting system (2017, background): “Confusion networks [25], and time-quantized lattices [26] are generated from the posterior lattices.”
Active learning for automatic speech recognition (2002, methodology): “’s algorithm to compute confusion networks (sausages) from the lattice output of a speech recognizer, and use the word posterior probability estimates on the sausages as word confidence scores [9].”
JHU Kaldi system for Arabic MGB-3 ASR challenge using diarization, audio-transcript alignment and transfer learning (2017, background): “Confusion network has been proven to be effective in dealing with uncertainties in various ASR applications [9].”
Enriching speech recognition with automatic detection of sentence boundaries and disfluencies (2006, methodology): “The resulting hypotheses were then combined into a single word confusion network [68],14 using only the recognition scores so that the single best word stream thus obtained was not changed by adding the SU events.”