Streaming End-to-end Speech Recognition for Mobile Devices

By Yanzhang He, Tara N. Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang and 10 more. 2018

676 citations recorded by the source corpus.

End-to-end (E2E) models, which directly predict output character sequences given input speech, are good candidates for on-device speech recognition. E2E models, however, present numerous challenges: In order to be truly useful, such models must decode speech utterances in a streaming fashion, in real time; they must be robust to the long tail of use cases; they must be able to leverage user-specific context (e.g., contact lists); and above all, they must be extremely accurate. In this work, we describe our efforts at building an E2E speech recog-nizer using a recurrent neural network transducer. In experimental evaluations, we find that the proposed approach can outperform a conventional CTC-based model in terms of both latency and accuracy in a number of evaluation categories.

Subfields: Artificial Intelligence, Efficiency & Compression, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

In Speech & Audio

In Signal Processing

Citations in the corpus

End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures (2019, neutral): “Nonetheless, there have been increasingly more end-to-end, competitive results, trained with CTC [7, 8], ASG [9], LF-MMI [10], sequence-to-sequence (Seq2Seq) [11, 12], or transducers training [13, 14], and differentiable decoding [15].”
BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition (2021, methodology): “RNN-T model of [81] with an 8-layer uni-directional LSTM decoder with cell size 2048, and a 2-layer LSTM decoder with the same cell size trained with the large multi-domain dataset”
Multitask Training with Text Data for End-to-End Speech Recognition (2020, background): “Attention-based end-to-end (E2E) speech recognition systems map audio features directly to text-level representations [1, 2, 3, 4, 5].”
Conformer: Convolution-augmented Transformer for Speech Recognition (2020, background): “Recurrent neural networks (RNNs) have been the defacto choice for ASR [1, 2, 3, 4] as they can model the temporal dependencies in the audio sequences effectively [5].”
ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context (2020, methodology): “We directly use the LSTM based label encoder and the joint network from [20], but propose a new CNN based audio encoder.”
Personalization of End-to-End Speech Recognition on Mobile Devices for Named Entities (2019, methodology): “A RNNT model has also been successfully deployed to run efficiently on mobile devices [3].”
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “We also use an advanced ASR model based on transducer [60] 6 .”
Specaugment on Large Scale Datasets (2019, methodology): “We train an RNN-T model described in [28].”
Transformer-Based Acoustic Modeling for Hybrid Speech Recognition (2019, background): “Among them, recurrent neural networks (RNNs), especially long short-term memory (LSTM) [7] neural networks, are widely used, either in conventional hybrid systems (e.g., [3, 8]), sequence-to-sequence-based (e.g. [9, 10]) or neural-transducer-based end-to-end systems (e.g. [11]).”
From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognition (2019, background): “It is also particularly interesting to compare chenones and word pieces [25, 26, 29] more closely.”
End-to-End Speech Recognition: A Survey (2023, methodology): “Inthetime-synchronous beam search, this fusion is performed by incorporating the language model score in the label transition state [22], [70], [258].”
An Investigation Into On-device Personalization of End-to-end Automatic Speech Recognition Models (2019, methodology): “Recently, an RNN-T ASR model has been developed for mobile devices and achieves state-of-the-art performance on a test set of Google’s voice search traffic [14].”
Improving Streaming Automatic Speech Recognition with Non-Streaming Model Distillation on Unsupervised Data (2020, background): “End-to-end streaming models [1, 2, 3, 4, 5, 6, 7] have become attractive for on-device recognition tasks in two aspects: first, endto-end models are usually compact, which makes them suitable to be used on devices.”
Two-Pass End-to-End Speech Recognition (2019, methodology): “Specifically, we prune first-pass beam candidates if they are too far in threshold from the current best candidate, where the threshold optimizes first-pass latency following [1].”
Comparing Supervised Models And Learned Speech Representations For Classifying Intelligibility Of Disordered Speech On Selected Phrases (2021, methodology): “[31] and has been trained with modifications on more diverse acoustic data to improve long-form speech recognition [2].”
Universal ASR: Unify and Improve Streaming ASR with Full-context Modeling (2020, methodology): “Sainath et al. further proposed to improve quality by using two-pass models (Sainath et al., 2019), i.e ., a second-pass LAS-based rescore model on top of the hypotheses from ﬁrst-pass RNN-T streaming output.”
Semi-Supervision in ASR: Sequential MixMatch and Factorized TTS-Based Augmentation (2021, methodology): “The training set is created by artiﬁcially corrupting clean utterances using a room simulator and addition of various noise styles [39].”