Exploring architectures, data and units for streaming end-to-end speech recognition with RNN-transducer

By Kanishka Rao, Hasim Sak, Rohit Prabhavalkar. 2017

365 citations recorded by the source corpus.

We investigate training end-to-end speech recognition models with the recurrent neural network transducer (RNN-T): a streaming, all-neural, sequence-to-sequence architecture which jointly learns acoustic and language model components from transcribed acoustic data. We explore various model architectures and demonstrate how the model can be improved further if additional text or pronunciation data are available. The model consists of an ‘encoder’, which is initialized from a connectionist temporal classification-based (CTC) acoustic model, and a ‘decoder’ which is partially initialized from a recurrent neural network language model trained on text data alone. The entire neural network is trained with the RNN-T loss and directly outputs the recognized transcript as a sequence of graphemes, thus performing end-to-end speech recognition. We find that performance can be improved further through the use of sub-word units ('wordpieces') which capture longer context and significantly reduce substitution errors. The best RNN-T system, a twelve-layer LSTM encoder with a two-layer LSTM decoder trained with 30,000 wordpieces as output targets achieves a word error rate of 8.5% on voice-search and 5.2% on voice-dictation tasks and is comparable to a state-of-the-art baseline at 8.3% on voice-search and 5.4% voice-dictation.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

In Signal Processing

Citations in the corpus

Unsupervised Speech Recognition (2021, background): “Such models are often classified into two categories: hybrid (Juang et al., 1986; Young, 1996; Povey, 2005; Hinton et al., 2012; Abdel-Hamid et al., 2012; Bourlard and Morgan, 2012) or end-to-end (Graves et al., 2006; Graves, 2012; Chorowski et al., 2015; Rao et al., 2017).”
Multitask Training with Text Data for End-to-End Speech Recognition (2020, background): “Attention-based end-to-end (E2E) speech recognition systems map audio features directly to text-level representations [1, 2, 3, 4, 5].”
Conformer: Convolution-augmented Transformer for Speech Recognition (2020, background): “Recurrent neural networks (RNNs) have been the defacto choice for ASR [1, 2, 3, 4] as they can model the temporal dependencies in the audio sequences effectively [5].”
ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context (2020, methodology): “Our network is based on the RNN-Transducer framework [18, 19, 20].”
Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss (2020, methodology): “(3), which are LSTMs in conventional RNN-T architectures [10, 12, 11], using the Transformers described above.”
State-of-the-Art Speech Recognition with Sequence-to-Sequence Models (2017, methodology): “[8] K. Rao, R. Prabhavalkar, and H. Sak, “Exploring Architectures, Data and Units for Streaming End-to-End Speech Recognition with RNN-Transducer,” in Proc.”
Personalization of End-to-End Speech Recognition on Mobile Devices for Named Entities (2019, background): “End-to-end speech recognition systems based on the Recurrent Neural Network Transducer architecture (RNN-T) [1] have been shown to achieve state-of-the-art performance for large-vocabulary continuous speech recognition [2].”
Streaming End-to-end Speech Recognition for Mobile Devices (2018, methodology): “Similar to [21], we train with word-piece subword units [26], which outperform graphemes in our experiments.”
An Analysis of Incorporating an External Language Model into a Sequence-to-Sequence Model (2017, methodology): “Following [26], the RNN LM is trained on about half a”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, methodology): “This ASR system can be either a conventional acoustic model in a hybrid-based ASR system, or an end-to-end speech recognition such as RNN-Transducers [34, 35] or Encoder-Decoder based model [36, 37].”
Model Unit Exploration for Sequence-to-Sequence Speech Recognition (2019, neutral): “, RNN-T [37, 38]) show similar trends.”
Bytes Are All You Need: End-to-end Multilingual Speech Recognition and Synthesis with Bytes (2018, methodology): “Sub-word units were used in seq2seq [12–14] and recurrent neural network transducer (RNNT) [15] models, and word units were used by [16, 17].”
Rnn-Transducer with Stateless Prediction Network (2020, methodology): “Additionally, the encoder-prediction structure of RNNT models has been understood to resemble the AM-LM structure in the classic ASR systems [2].”
Acoustically Grounded Word Embeddings for Improved Acoustics-to-word Speech Recognition (2019, methodology): “However, to achieve performance comparable with traditional methods, these systems are still typically trained to predict sub-word units such as characters or “wordpieces” [1–4], thereby relying on additional decoders and externally trained language models.”
Recognizing Long-Form Speech Using Streaming End-to-End Models (2019, background): “On medium-scale tasks such as Librispeech [15] or largescale voice-search tasks [16, 17, 18], such E2E models have been shown to outperform conventional ASR systems.”
A Streaming On-Device End-To-End Model Surpassing Server-Side Conventional Model Quality and Latency (2020, background): “End-to-end (E2E) models [2, 3, 4, 5, 6, 7, 8, 9] have gained large popularity in the automatic speech recognition (ASR) community over the last few years.”
An Investigation Into On-device Personalization of End-to-end Automatic Speech Recognition Models (2019, background): “Recently, the recurrent neural network transducer (RNN-T) [19] and Listen-Attend-Spell (LAS) [20] end-to-end ASR models have been successfully applied to large vocabulary continuous speech recognition [21, 22].”