Comparing Human and Machine Errors in Conversational Speech Transcription

By Andreas Stolcke, Jasha Droppo. 2017

72 citations recorded by the source corpus.

Recent work in automatic recognition of conversational telephone speech (CTS) has achieved accuracy levels comparable to human transcribers, although there is some debate how to precisely quantify human performance on this task, using the NIST 2000 CTS evaluation set. This raises the question what systematic differences, if any, may be found differentiating human from machine transcription errors. In this paper we approach this question by comparing the output of our most accurate CTS recognition system to that of a standard speech transcription vendor pipeline. We find that the most frequent substitution, deletion and insertion error types of both outputs show a high degree of overlap. The only notable exception is that the automatic recognizer tends to confuse filled pauses ("uh") and backchannel acknowledgments ("uhhuh"). Humans tend not to make this error, presumably due to the distinctive and opposing pragmatic functions attached to these words. Furthermore, we quantify the correlation between human and machine errors at the speaker level, and investigate the effect of speaker overlap between training and test data. Finally, we report on an informal "Turing test" asking humans to discriminate between automatic and human transcription error cases.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Toward Domain-Invariant Speech Recognition via Large Scale Training (2018, background): “Automatic speech recognition (ASR) has come a long way in the last few years, with state-of-the-art systems performing close to human performance [1, 2].”
The CAPIO 2017 Conversational Speech Recognition System (2017, background): “Achieving human parity has now become the topic of the speech recognition community, which nurtured interesting research works of contrasting transcriptions from human transcribers and conversational speech recognition systems [1, 2, 6].”
Deep convolutional neural networks for eigenvalue problems in mechanics (2018, background): “The latter techniques have recently reached human-level accuracy [8, 9].”
Toward Human Parity in Conversational Speech Recognition (2017, methodology): “which significantly boosts our bidirectional long shortterm memory (BLSTM) acoustic model performance 3) The use of long short-term memory (LSTM) rather than recurrent neural nets (RNNs) as language models, and the use of a letter-trigram input representation 4) A two-level system combination, based”
The Microsoft 2017 Conversational Speech Recognition System (2017, background): “The additional knowledge source at this stage was motivated by our analysis of differences between machine versus human transcription errors [24].”