Recognizing Long-Form Speech Using Streaming End-to-End Models

By Arun Narayanan, Rohit Prabhavalkar, Chung-Cheng Chiu, David Rybach, Tara N. Sainath, Trevor Strohman. 2019

137 citations recorded by the source corpus.

All-neural end-to-end (E2E) automatic speech recognition (ASR) systems that use a single neural network to transduce audio to word sequences have been shown to achieve state-of-the-art results on several tasks. In this work, we examine the ability of E2E models to generalize to unseen domains, where we find that models trained on short utterances fail to generalize to long-form speech. We propose two complementary solutions to address this: training on diverse acoustic data, and LSTM state manipulation to simulate long-form audio when training using short utterances. On a synthesized long-form test set, adding data diversity improves word error rate (WER) by 90% relative, while simulating long-form training improves it by 67% relative, though the combination doesn't improve over data diversity alone. On a real long-form call-center test set, adding data diversity improves WER by 40% relative. Simulating long-form training on top of data diversity improves performance by an additional 27% relative.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Tara N. Sainath
3. Yu Zhang
4. Dong Yu
5. Yonghui Wu
6. Chung-Cheng Chiu
7. Bo Li
8. Rohit Prabhavalkar

In Signal Processing

1. Jinyu Li
2. Tara N. Sainath
3. Dong Yu
4. Yu Zhang
5. Chung-Cheng Chiu
6. Yonghui Wu
7. Rohit Prabhavalkar
8. Bo Li

Citations in the corpus

• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, background): “main source, but the massive audiobook data mismatches the data in a real scenario and using it exclusively hurts the model performance when the acoustic characteristics of the downstream tasks are different from those of the audiobook [10]–[13].”
• A Better and Faster end-to-end Model for Streaming ASR (2020, methodology): “To address this, we look at running a 2nd-pass beam search with RNN-T itself, which has shown better robustness to long-form [26].”
• Improving Streaming Automatic Speech Recognition with Non-Streaming Model Distillation on Unsupervised Data (2020, methodology): “variational weight noise, SpecAugment [21], and random state sampling and random state passing [18].”
• Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “AED models also cannot perform well on long utterances [54, 55].”
• Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models (2021, methodology): “Both YT-CTC and YT-RNNT models are trained from YouTube data, whereas MD-RNNT is trained on multi-domain data [24].”
• Context-Aware end-to-end ASR Using Self-Attentive Embedding and Tensor Fusion (2023, methodology): “Among the E2E models, the Recurrent Neural Network Trans-ducer (RNN-T) provides good accuracy, robustness to longform [9] and low latency [10], and thus will be the model used in this work.”
• An Attention-Based Joint Acoustic and Text on-Device End-To-End Model (2020, methodology): “Similar to [25], features for each frame are stacked with 3 frames to the left and then downsampled by three to a 30ms frame rate.”
• Less is More: Improved RNN-T Decoding Using Limited Label Context and Path Merging (2020, methodology): “Note that this can have a large impact since end-to-end models are typically decoded with small number of candidates in the beam for efficiency [12], and thus the beam diversity tends to reduce for longer utterances [27].”
• A Streaming On-Device End-To-End Model Surpassing Server-Side Conventional Model Quality and Latency (2020, methodology): “First, on the quality-front, we train our model on multi-domain audio-text utterance pairs, utilizing sources from different domains including search traffic, telephony data and YouTube data [1].”
• RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions (2020, methodology): “We further evaluate the model on a long-form call-center test set described in [19] to assess its robustness on unseen domain.”
• Comparing Supervised Models And Learned Speech Representations For Classifying Intelligibility Of Disordered Speech On Selected Phrases (2021, methodology): “• Classifiers trained on representations extracted from the LSTM encoder of a production quality ASR model [2].”