A Streaming On-Device End-To-End Model Surpassing Server-Side Conventional Model Quality and Latency

By Tara N. Sainath, Yanzhang He, Bo Li, Arun Narayanan, Ruoming Pang, Antoine Bruguier, Shuo-yiin Chang, Wei Li, Raziel Alvarez, Z. Chen and 19 more. 2020

228 citations recorded by the source corpus.

Thus far, end-to-end (E2E) models have not been shown to outperform state-of-the-art conventional models with respect to both quality, i.e., word error rate (WER), and latency, i.e., the time the hypothesis is finalized after the user stops speaking. In this paper, we develop a first-pass Recurrent Neural Network Transducer (RNN-T) model and a second-pass Listen, Attend, Spell (LAS) rescorer that surpasses a conventional model in both quality and latency. On the quality side, we incorporate a large number of utterances across varied domains [1] to increase acoustic diversity and the vocabulary seen by the model. We also train with accented English speech to make the model more robust to different pronunciations. In addition, given the increased amount of training data, we explore a varied learning rate schedule. On the latency front, we explore using the end-of-sentence decision emitted by the RNN-T model to close the microphone, and also introduce various optimizations to improve the speed of LAS rescoring. Overall, we find that RNN-T+LAS offers a better WER and latency tradeoff compared to a conventional model. For example, for the same latency, RNN-T+LAS obtains a 8% relative improvement in WER, while being more than 400-times smaller in model size.

Subfields: Artificial Intelligence, Efficiency & Compression, ML Systems & Hardware, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Tara N. Sainath
3. Dong Yu
4. Yu Zhang
5. Shinji Watanabe
6. Chung-Cheng Chiu
7. Yonghui Wu
8. Bo Li

In Efficiency & Compression

1. Jinyu Li
2. Tara N. Sainath
3. Dong Yu
4. Chung-Cheng Chiu
5. Yonghui Wu
6. Yu Zhang
7. Jiahui Yu
8. Wei Han

In ML Systems & Hardware

1. Shinji Watanabe
2. Bo Li
3. Dong Yu
4. Jinyu Li
5. Tara N. Sainath
6. Shibo Wang
7. Yu Zhang
8. Yongqiang Wang

In Signal Processing

1. Jinyu Li
2. Dong Yu
3. Tara N. Sainath
4. Yu Zhang
5. Shinji Watanabe
6. Chung-Cheng Chiu
7. Bo Li
8. Rohit Prabhavalkar

Citations in the corpus

• Conformer: Convolution-augmented Transformer for Speech Recognition (2020, background): “Recurrent neural networks (RNNs) have been the defacto choice for ASR [1, 2, 3, 4] as they can model the temporal dependencies in the audio sequences effectively [5].”
• ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context (2020, methodology): “For instance, we use a RNN-T decoder [18, 19, 20, 21] instead of the CTC decoder [22].”
• End-to-End Speech Recognition: A Survey (2023, methodology): “The first streaming E2E model, deployed to production, was launched in 2019 for the Pixel 4 smartphone [22], [332].”
• Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models (2021, background): “Recent research has focused on improving streaming ASR by trading off latency [11, 3, 12].”
• Improving Streaming Automatic Speech Recognition with Non-Streaming Model Distillation on Unsupervised Data (2020, background): “In the past few years, many research efforts have been devoted to improving streaming ASR [10, 11, 12].”
• Fast, Simpler and More Accurate Hybrid ASR Systems Using Wordpieces (2020, background): “Most recently, neural transducers [24] have shown great potential for both on-device [25] and server [26] use cases.”
• Universal ASR: Unify and Improve Streaming ASR with Full-context Modeling (2020, background): “Improving both WER and emission latency has been shown to be highly challenging (He et al., 2019; Li et al., 2020a; Sainath et al., 2020) in streaming ASR systems.”
• Self-supervised Learning with Random-projection Quantizer for Speech Recognition (2022, background): “In addition to the non-streaming models, streaming architecture also plays a critical role for the speech recognition tasks as many of the applications require transcribing speakers’ utterances with low-latency (Sainath et al., 2020).”
• s-Transformer: Segment-Transformer for Robust Neural Speech Synthesis (2020, methodology): “In the speech recognition tasks, there are many works on the streaming end-to-end model [14], using CTC plus Transformer [15][16] or RNN-Transducer [17] plus LAS [18] [19].”
• Speechmoe2: Mixture-of-Experts Model with Improved Routing (2021, background): “[13] adds global domain information to the end-to-end models and shows that feeding the domain-id improves the model performance.”
• Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “Therefore, E2E models have been shown to outperform traditional hybrid models not only in academics [10] but also in the industry [11, 12].”
• Less is More: Improved RNN-T Decoding Using Limited Label Context and Path Merging (2020, methodology): “Model Architecture: Our experimental setup is similar to our previous work [12].”
• Continuous Speech Separation with Conformer (2020, unknown): “former 1. INTRODUCTION Thanks to the advance in deep learning, a drastic improvement on accuracy and robustness has been brought to modern automatic speech recognition (ASR) system in the past decade [1, 2, 3, 4, 5, 6, 7, 8]. However, when applied to more complicated scenarios such as conversation t”
• Fast and Parallel Decoding for Transducer (2022, unknown): “…architecture [1] has been growing in popularity in the field of automatic speech recognition (ASR), especially for deployed real-time ASR systems [2, 3] because it support streaming naturally while achieving high accuracy Unlike decoding with CTC models where at most one symbol can be emitted…”