Toward Human Parity in Conversational Speech Recognition

By Wayne Xiong, Jasha Droppo, Xuedong Huang, Frank Torsten Bernd Seide, M. Seltzer, Andreas Stolcke, Dong Yu, Geoffry Zweig. 2017

200 citations recorded by the source corpus.

Conversational speech recognition has served as a flagship speech recognition task since the release of the Switchboard corpus in the 1990s. In this paper, we measure a human error rate on the widely used NIST 2000 test set for commercial bulk transcription. The error rate of professional transcribers is 5.9% for the Switchboard portion of the data, in which newly acquainted pairs of people discuss an assigned topic, and 11.3% for the CallHome portion, where friends and family members have open-ended conversations. In both cases, our automated system edges past the human benchmark, achieving error rates of 5.8% and 11.0%, respectively. The key to our system's performance is the use of various convolutional and long-short-term memory acoustic model architectures, combined with a novel spatial smoothing method and lattice-free discriminative acoustic training, multiple recurrent neural network language modeling approaches, and a systematic use of system combination. Comparing frequent errors in our human and machine transcripts, we find them to be remarkably similar, and highly correlated as a function of the speaker. Human subjects find it very difficult to tell which errorful transcriptions come from humans. Overall, this suggests that, given sufficient matched training data, conversational speech transcription engines are approximating human parity in both quantitative and qualitative terms.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Tie-Yan Liu
3. Xu Tan
4. Yingce Xia
5. Lijun Wu
6. Fei Tian
7. Furu Wei
8. Wei-Wei Tu

In Speech & Audio

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Sheng Zhao
5. Xuedong Huang
6. Shujie Liu
7. Qiang Yang
8. Frank Torsten Bernd Seide

In Signal Processing

1. Xu Tan
2. Tie-Yan Liu
3. Tao Qin
4. Thorsten Holz
5. Xuedong Huang
6. Yang Yu
7. Qiang Yang
8. Dorothea Kolossa

Citations in the corpus

• CHiME-6 Challenge: Tackling Multispeaker Speech Recognition for Unsegmented Recordings (2020, neutral): “In parallel to research on acoustic robustness, research on conversational speech recognition has also made great progress, as illustrated by the recent announcements of super-human performance [22, 23] achieved on the Switchboard telephone conversation task [24] and by the ASpIRE challenge [25].”
• Achieving Human Parity on Automatic Chinese to English News Translation (2018, background): “Recent years have seen human performance levels reached or surpassed in tasks ranging from games such as Go [32] to classification of images in ImageNet [20] to conversational speech recognition on the Switchboard task [49].”
• Adversarial Attacks Against Automatic Speech Recognition Systems via Psychoacoustic Hiding (2018, background): “Deep neural networks (DNNs) have evolved into the state-of-the-art approach for many machine learning tasks, including automatic speech recognition (ASR) systems [59], [45].”
• Taking Human out of Learning Applications: A Survey on Automated Machine Learning (2018, background): “For example, AlphaGO [2] beat human champion in the game of GO, ResNet [3] surpassed human performance in image recognition, Microsoft’s speech system [4] approximated human level in speech transcription.”
• An anatomy for neural search engines (2019, background): “For a little over a decade, the deep learning revolution has been dramatically improving how computers perform on a variety of tasks, very often surpassing human capabilities [He et al., 2015; Xiong et al., 2017b; Silver et al., 2017].”
• Artificial intelligence in radiology (2018, unknown): “Currently , we are witnessing narrow task-specific AI applications that are able to match and occasionally surpass human intelligence 4–6 , 9 . are vital.”