Building Competitive Direct Acoustics-to-Word Models for English Conversational Speech Recognition

By Kartik Audhkhasi, Brian Kingsbury, Bhuvana Ramabhadran, George Saon, Michael A. Picheny. 2017

154 citations recorded by the source corpus.

Direct acoustics-to-word (A2W) models in the end-to-end paradigm have received increasing attention compared to conventional subword based automatic speech recognition models using phones, characters, or context-dependent hidden Markov model states. This is because A2W models recognize words from speech without any decoder, pronunciation lexicon, or externally-trained language model, making training and decoding with such models simple. Prior work has shown that A2W models require orders of magnitude more training data in order to perform comparably to conventional models. Our work also showed this accuracy gap when using the English Switchboard-Fisher data set. This paper describes a recipe to train an A2W model that closes this gap and is at-par with state-of-the-art sub-word based models. We achieve a word error rate of 8.8.8%/13.9% on the Hub5-2000 Switchboard/CallHome test sets without any decoder or language model. We find that model initialization, training data order, and regularization have the most impact on the A2W model performance. Next, we present a joint word-character A2W model that learns to first spell the word and then recognize it. This model provides a rich output to the user instead of simple word hypotheses, making it especially useful in the case of words unseen or rarely-seen during training.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition (2019, neutral): “[30] 9.6 19.3 Hadian et al., (2018) [42] 9.3 18.9 Zeyer et al., (2018) [24] 8.3 17.3 CTC Zweig et al., (2017) [43] 24.7 37.1 14.0 25.3 Audhkhasi et al., (2018) [44] 20.8 30.4 Audhkhasi et al., (2018) [45] 14.6 23.6 LAS Lu et al., (2016) [46] 26.8 48.2 25.8 46.0 Toshniwal et al., (2017) [47] 23.1 40.”
Speech Model Pre-training for End-to-End Spoken Language Understanding (2019, methodology): “ASR models are trained using a variety of targets, including phonemes, graphemes, wordpieces, or more recently whole words [28–30].”
Acoustically Grounded Word Embeddings for Improved Acoustics-to-word Speech Recognition (2019, methodology): “Other recent work [6,7,9] has explored techniques for training on much less data (e.g., 300 hours), but performance gaps still remain between A2W and sub-word models.”
From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognition (2019, background): “, “the cat”) [20, 36, 37], or a mixture of words and graphemes [38, 39].”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “Both of these approaches to end-to-end ASR usually predict a sequence of characters, although there have also been initial attempts at directly predicting words [13, 313].”
Improved Knowledge Distillation from Bi-Directional to Uni-Directional LSTM CTC for End-to-End Speech Recognition (2018, background): “Instead, E2E ASR only requires pairs of input feature sequences and output symbol sequences [5, 6], such as phones [7], characters [8], words [9, 10], or their combinations [11, 12].”
Combining Residual Networks with LSTMs for Lipreading (2017, background): “Nevertheless, recent advances in speech recognition and natural language processing show that direct modeling of words is feasible even for LVCSR, [7] [8] [9].”
Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “While most units (characters [2, 3], word-piece [262–266], words [18, 22, 267], or even phrases [268]) are purely derived based on text and word-piece unit is the dominating one, there are also studies working on phoneme units [269–272].”
Recognizing Long-Form Speech Using Streaming End-to-End Models (2019, background): “Examples include connectionist temporal classification (CTC) with character [3, 4] or wordbased [5, 6] targets, the recurrent neural network transducer (RNNT) [7, 8], and attention-based encoder-decoder models [9, 10, 11, 12, 13, 14].”
Minimum Bayes Risk Training of RNN-Transducer for End-to-End Speech Recognition (2019, background): “There are mainly three categories of end-to-end speech recognition systems, namely, connectionist temporal classification (CTC) based [1–6], attention based [7–9] and RNN-T based [10–14].”
RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions (2020, background): “, connectionist temporal classification (CTC) [2, 3] with character [4, 5] or word [6, 7] targets; attention-based encoder-decoder models [8, 9, 10, 11, 12]; and the recurrent neural network transducer (RNN-T) [13, 14, 15].”
Hierarchical Multitask Learning for CTC-based Speech Recognition (2018, background): “For example, Audhkhasi et al. [15, 30] re-ported signiﬁcant improvements by pretraining a whole-word BiL-STM CTC network with a phone CTC loss, as well as more modest improvements in a hierarchical setting where all but the last BiL-STM layer were pretrained and the full model was then trained with…”
Recent progresses in deep learning based acoustic models (2017, methodology): “In [71], [72], the displayed hypothesis is more meaningful than OOV to users.”
Adversarial Attacks Against Automatic Speech Recognition Systems via Psychoacoustic Hiding (2018, methodology): “We evaluated it against the state-of-the-art DNN-HMM-based ASR system Kaldi [40], which is one of the most popular toolchains for ASR among researchers [1], [19], [29], [30], [42], [43], [52], [53], [55], [61] and is also used in commercial products such as Amazon’s Echo/Alexa and by IBM and Microso”
A Comparison of Transformer and LSTM Encoder Decoder Models for ASR (2019, unknown): “End-to-end automatic speech recognition (ASR) tries to simplify the modeling aspect, the training pipeline, and the decoding algorithm [1–8].”