Direct Acoustics-to-Word Models for English Conversational Speech Recognition

By Kartik Audhkhasi, Bhuvana Ramabhadran, George Saon, Michael A. Picheny, David Nahamoo. 2017

71 citations recorded by the source corpus.

Recent work on end-to-end automatic speech recognition (ASR) has shown that the connectionist temporal classification (CTC) loss can be used to convert acoustics to phone or character sequences. Such systems are used with a dictionary and separately-trained Language Model (LM) to produce word sequences. However, they are not truly end-to-end in the sense of mapping acoustics directly to words without an intermediate phone representation. In this paper, we present the first results employing direct acoustics-to-word CTC models on two well-known public benchmark tasks: Switchboard and CallHome. These models do not require an LM or even a decoder at run-time and hence recognize speech with minimal complexity. However, due to the large number of word output units, CTC word models require orders of magnitude more data to train reliably compared to traditional systems. We present some techniques to mitigate this issue. Our CTC word model achieves a word error rate of 13.0%/18.8% on the Hub5-2000 Switchboard/CallHome test sets without any LM or decoder compared with 9.6%/16.0% for phone-based CTC with a 4-gram LM. We also present rescoring results on CTC word model lattices to quantify the performance benefits of a LM, and contrast the performance of word and phone CTC models.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition (2019, neutral): “, (2018) [44] 20.”
Improved training of end-to-end attention models for speech recognition (2018, methodology): “Connectionist temporal classiﬁcation (CTC) [10] has been often used as an end-to-end model for speech recognition, often on characters/graphemes [11–16] or on sub-words [17] but also directly on words [18,19].”
Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems (2017, background): “Both these of these approaches to end-to-end ASR usually predict a sequence of characters, although there have also been initial attempts at directly predicting words [9, 10].”
Improved training for online end-to-end speech recognition systems (2017, methodology): “In case of the grapheme-level or word-level CTC models, it was shown that starting from phoneme-based model was essential in obtaining better performance [18–20].”
Building Competitive Direct Acoustics-to-Word Models for English Conversational Speech Recognition (2017, methodology): “[20] with the following changes compared to [17]: • We included delta and delta-delta coefﬁcients because they slightly improved the WER.”
Large Margin Training for Attention Based End-to-End Speech Recognition (2019, background): “It is observed from prior works that decoding end-to-end speech recognition systems with shallow-fused external language models can improve the recognition performances [24, 25, 26].”
Recent progresses in deep learning based acoustic models (2017, methodology): “In [18], [62], [70], only the most frequent words in the training set were used as targets whereas the remaining words were just tagged as OOVs.”
Word-Level Speech Recognition With a Letter to Word Encoder (2019, background): “Prior work has also attempted direct-to-word speech recognition (Audhkhasi et al., 2017; Li et al., 2018; Soltau et al., 2016).”