Multiple Dimension Levenshtein Edit Distance Calculations for Evaluating Automatic Speech Recognition Systems During Simultaneous Speech

By Jonathan G. Fiscus, Jerome Ajot, Nicolas Radde, Christophe Laprun. 2006

61 citations recorded by the source corpus.

Since 1987, the National Institute of Standards and Technology has been providing evaluation infrastructure for the Automatic Speech Recognition (ASR), and more recently referred to as the Speech-To-Text (STT), research community. From the first efforts in the Resource Management domain to the present research, the NIST SCoring ToolKit (SCTK) has formed the tool set for system developers to make continued progress in many domains; Wall Street Journal, Conversational Telephone Speech (CTS), Broadcast News (BN), and Meetings (MTG) to name a few. For these domains, the community agreed to declared sections of simultaneous speech as “not scoreable”. While this had minor impact on most of these domains, the highly interactive nature of Meeting speech rendered a very large fraction of the test material not scoreable. This paper documents a multi-dimensional extension of the Dynamic Programming solution to Levenshtein Edit Distance calculations capable of evaluating STT systems during periods of overlapping, simultaneous speech.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Juan Pino
2. Changhan Wang

In Evaluation & Benchmarks

1. Francis M. Tyers
2. Deliang Wang
3. Yuxuan Wang

In Speech & Audio

1. Joshua D. Eisenberg
2. Lindy Rauchenstein
3. Josh Meyer
4. Nicholas Howell
5. Juan Pino
6. Changhan Wang

In Signal Processing

1. Changhan Wang
2. Juan Pino

Citations in the corpus

• Common Voice: A Massively-Multilingual Speech Corpus (2019, methodology): “Each cell in the table contains the Character Error Rate (CER)12 of the resulting model on the test set, defined as the Levenshtein distance (Fiscus et al., 2006) of the characters between the ground-truth transcript and the decoding result.”
• Artie Bias Corpus: An Open Dataset for Detecting Demographic Bias in Speech Applications (2020, methodology): “5.1 ter Error Rate (CER)11, computed as percentage of incor rectly transcribed characters as per the Levenshtein distance between the source transcript and the predicted transcript (Fiscus et al., 2006).”
• The Rich Transcription 2007 Meeting Recognition Evaluation (2007, methodology): “We used ASCLITE’s [8] Multiple System Streamto-Multiple Reference Stream alignment capabilities to compute the word alignments.”