English Conversational Telephone Speech Recognition by Humans and Machines

By George Saon, Gakuto Kurata, Tom Sercu, Kartik Audhkhasi, Samuel Thomas, Dimitrios Dimitriadis, Xiaodong Cui, Bhuvana Ramabhadran, Michael A. Picheny, L. Lim and 2 more. 2017

374 citations recorded by the source corpus.

One of the most difficult speech recognition tasks is accurate recognition of human to human communication. Advances in deep learning over the last few years have produced major speech recognition improvements on the representative Switchboard conversational corpus. Word error rates that just a few years ago were 14% have dropped to 8.0%, then 6.6% and most recently 5.8%, and are now believed to be within striking range of human performance. This then raises two issues - what IS human performance, and how far down can we still drive speech recognition error rates? A recent paper by Microsoft suggests that we have already achieved human performance. In trying to verify this statement, we performed an independent set of human performance measurements on two conversational tasks and found that human performance may be considerably better than what was earlier reported, giving the community a significantly harder goal to achieve. We also report on our own efforts in this area, presenting a set of acoustic and language modeling techniques that lowered the word error rate of our own English conversational telephone LVCSR system to the level of 5.5%/10.3% on the Switchboard/CallHome subsets of the Hub5 2000 evaluation, which - at least at the writing of this paper - is a new performance milestone (albeit not at what we measure to be human performance!). On the acoustic side, we use a score fusion of three models: one LSTM with multiple feature inputs, a second LSTM trained with speaker-adversarial multi-task learning and a third residual net (ResNet) with 25 convolutional layers and time-dilated convolutions. On the language modeling side, we use word and character LSTMs and convolutional WaveNet-style language models.

Subfields: Artificial Intelligence, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

Citations in the corpus

End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures (2019, neutral): “, 2016) and have since been successfully applied to speech recognition (Xiong et al., 2017; Saon et al., 2017; Li et al., 2019b; Wang et al., 2017).”
GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10, 000 Hours of Transcribed Audio (2021, background): “5% on the Switchboard portion of the Hub5’00 benchmark [5].”
Toward Domain-Invariant Speech Recognition via Large Scale Training (2018, background): “Automatic speech recognition (ASR) has come a long way in the last few years, with state-of-the-art systems performing close to human performance [1, 2].”
Opportunities and obstacles for deep learning in biology and medicine (2017, background): “Since 2009 there have been drastic performance improvements with error rates dropping from more than 20% to less than 6% [422] and finally approaching or exceeding human performance in the past year [423,424].”
The CAPIO 2017 Conversational Speech Recognition System (2017, background): “With the contributions mainly from convolutional neural networks (CNNs) and recurrent neural networks (RNNs), multiple research groups have continued to improve their system performance on the well-known, industry-standard NIST 2000 Hub5 English evaluation set(1) [1, 2, 3, 4, 5], approaching to the ”
Multitalker Speech Separation With Utterance-Level Permutation Invariant Training of Deep Recurrent Neural Networks (2017, background): “More recently, a large number of techniques based on deep learning [19] have been proposed, especially for Automatic Speech Recognition (ASR) [20]–[25], and speech enhancement [26]–[34].”
Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner (2016, neutral): “ later, the entire speech processing pipeline has been replaced by neural networks trained endto-end, with performance claimed to achieve human parity on a dictation task (Xiong et al., 2016, but see Saon et al., 2017). In the following, we very brieﬂy review how such systems are constructed before ”
Cycle-Consistent Speech Enhancement (2018, methodology): “In speech area, it has been applied to speech enhancement [24, 25, 26, 27], voice conversion [28, 29], acoustic model adaptation [30, 31, 32], noise-robust [33, 34] and speaker-invariant [35, 36] ASR using gradient reversal layer (GRL) [37].”
LSTM Language Models for LVCSR in First-Pass Decoding and Lattice-Rescoring (2019, background): “In recent years, language models (LMs) based on long short-term memory (LSTM) neural networks have become an integral part of many state-of-the-art automatic speech recogition systems [1, 2, 3].”
RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition (2018, background): “(Saon et al., 2017). hybrid 2 trained with Lattice-free MMI (Hadian et al., 2018).”
German End-to-end Speech Recognition based on DeepSpeech (2019, background): “This includes commercial services like Google Cloud Speech-to-Text (He et al., 2018), Amazon Alexa Voice Services 1 , IBM Watson Speech to Text (Saon et al., 2017) or Speechmatics 2 as well as academic services like BAS.”
Adversarial Attacks Against Automatic Speech Recognition Systems via Psychoacoustic Hiding (2018, background): “Deep neural networks (DNNs) have evolved into the state-of-the-art approach for many machine learning tasks, including automatic speech recognition (ASR) systems [59], [45].”
Direct Acoustics-to-Word Models for English Conversational Speech Recognition (2017, methodology): “The ”small” 4-gram LM used 24M words from the 2000-hour audio training data with a vocabulary size of 30k words, while the ”large” 4-gram LM used a vocabulary of 85k words with an additional 560M words from several public text data sets from LDC [2].”
Recent progresses in deep learning based acoustic models (2017, background): “Recently, the concept of adversarial training [147] was explored for noise-robust ASR [148]–[155] and speaker invariant training [156], [157].”
To Reverse the Gradient or Not: an Empirical Comparison of Adversarial and Multi-task Learning in Speech Recognition (2018, methodology): “This approach has been previously used for speech recognition to learn invariance to noise conditions [9] and speaker identity [10, 11, 12].”
A Review of Speaker Diarization: Recent Advances with Deep Learning (2021, background): “However, recent advances in the areas of Speech Recognition [200, 201], far-field speech processing [202, 203, 204], Speaker ID and diarization [205, 206, 113],”