Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition

By Hagen Soltau, Hank Liao, Hasim Sak. 2016

317 citations recorded by the source corpus.

We present results that show it is possible to build a competitive, greatly simplified, large vocabulary continuous speech recognition system with whole words as acoustic units. We model the output vocabulary of about 100,000 words directly using deep bi-directional LSTM RNNs with CTC loss. The model is trained on 125,000 hours of semi-supervised acoustic training data, which enables us to alleviate the data sparsity problem for word models. We show that the CTC word models work very well as an end-to-end all-neural speech recognition model without the use of traditional context-dependent sub-word phone units that require a pronunciation lexicon, and without any language model removing the need to decode. We demonstrate that the CTC word models perform better than a strong, more complex, state-of-the-art baseline with sub-word units.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Dong Yu
3. Tara N. Sainath
4. Yu Zhang
5. Shinji Watanabe
6. Yoshua Bengio
7. Li Deng
8. Quoc V. Le

In Speech & Audio

1. Jinyu Li
2. Tara N. Sainath
3. Yu Zhang
4. Shinji Watanabe
5. Dong Yu
6. Yonghui Wu
7. Wei-Ning Hsu
8. Chung-Cheng Chiu

In Signal Processing

1. Jinyu Li
2. Dong Yu
3. Tara N. Sainath
4. Shinji Watanabe
5. Yu Zhang
6. Rohit Prabhavalkar
7. Li Deng
8. Bo Li

Citations in the corpus

• Mixed Precision Training (2017, methodology): “For example, the speech recognition model used in Hannun et al. (2014) was trained on 5,000 hours of data, more recent acoustic models trained on nearly 12,000 hours (Amodei et al., 2016) and even larger data sets like 125,000 hours used in Soltau et al. (2016).”
• End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures (2019, neutral): “In ASR, semi-supervised pseudo-label-style self-training has been explored generally in end-to-end settings in (Soltau et al., 2016; Li et al., 2019a; Kahn et al., 2019a) for both low-resource (Veselỳ et al.”
• Toward Domain-Invariant Speech Recognition via Large Scale Training (2018, methodology): “The YouTube set was transcribed in a semi-supervised fashion [28, 29]; the rest of the datasets are hand-transcribed.”
• Lessons from Building Acoustic Models with a Million Hours of Speech (2019, background): “Building an AM from a hundred thousand hours is still rare, but [15] showed that increasing from several thousand hours of training data to a hundred thousand hours of lightly supervised data can yield”
• Streaming End-to-end Speech Recognition for Mobile Devices (2018, background): “∗Equal contribution Early E2E work examined connectionist temporal classification (CTC) [15] with grapheme or word targets [16, 17, 18, 19].”
• Improved training of end-to-end attention models for speech recognition (2018, methodology): “Connectionist temporal classification (CTC) [10] has been often used as an end-to-end model for speech recognition, often on characters/graphemes [11–16] or on sub-words [17] but also directly on words [18, 19].”
• English Conversational Telephone Speech Recognition by Humans and Machines (2017, methodology): “One possibility was to train an end-to-end LSTM using CTC as in [14, 15, 16, 17] and append the features from the last LSTM layer before the softmax to our existing features.”
• Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems (2017, background): “Both these of these approaches to end-to-end ASR usually predict a sequence of characters, although there have also been initial attempts at directly predicting words [9, 10].”
• Bytes Are All You Need: End-to-end Multilingual Speech Recognition and Synthesis with Bytes (2018, methodology): “Sub-word units were used in seq2seq [12–14] and recurrent neural network transducer (RNNT) [15] models, and word units were used by [16, 17].”
• Improved training for online end-to-end speech recognition systems (2017, methodology): “In case of the grapheme-level or word-level CTC models, it was shown that starting from phoneme-based model was essential in obtaining better performance [18–20].”
• Improved Knowledge Distillation from Bi-Directional to Uni-Directional LSTM CTC for End-to-End Speech Recognition (2018, background): “Instead, E2E ASR only requires pairs of input feature sequences and output symbol sequences [5, 6], such as phones [7], characters [8], words [9, 10], or their combinations [11, 12].”
• Time-Dependent Representation for Neural Event Sequence Prediction (2017, background): “…especially with Long-Short Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) have become popular in solving a variety of sequence prediction problems, including neural machine translation (Bahdanau et al., 2014), image captioning (Xu et al., 2015) and speech recognition (Soltau et al., 2016).”
• Hybrid CTC/Attention Architecture for End-to-End Speech Recognition (2017, unknown): “CTC permits the efficient computation of a strictly monotonic alignment using dynamic programming [15], [8] although it requires separate language models and graph-based decoding [16], except in the case of huge training data [17], [18].”
• Multimodal Language Analysis with Recurrent Multistage Fusion (2018, methodology): “…Networks (Elman, 1990; Jain and Medsker, 1999) have been used extensively for language and speech based sequence modeling (Zilly et al., 2016; Soltau et al., 2016), sentiment analysis (Socher et al., 2013; dos Santos and Gatti, 2014; Glorot et al., 2011; Cambria, 2016), and emotion…”
• From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognition (2019, background): “, “the cat”) [20, 36, 37], or a mixture of words and graphemes [38, 39].”
• VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking (2018, methodology): “The architecture of the Voice-Filter system is shown in Table 1, with a few different variations of the LSTM layer: (1) no LSTM layer, i.e. , only convolutional layers directly followed by fully connected layers; (2) a uni-directional LSTM layer; (3) a bi-directional LSTM layer.”
• Hybrid Autoregressive Transducer (HAT) (2020, methodology): “The letter models [19, 20, 21] and word model [22] are examples of this neural model class.”
• Recognizing Long-Form Speech Using Streaming End-to-End Models (2019, background): “The training sets include data from four domains: anonymized and hand-transcribed utterances representative of voice-search and far-field use cases, segmented telephony speech, and semi-supervised YouTube video segments [40, 5].”