RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions

By Chung-Cheng Chiu, Arun Narayanan, Wei Han, Rohit Prabhavalkar, Yu Zhang, Navdeep Jaitly, Ruoming Pang, Tara N. Sainath, Patrick Nguyen, Liangliang Cao and 1 more. 2020

44 citations recorded by the source corpus.

In recent years, all-neural end-to-end approaches have obtained state-of-the-art results on several challenging automatic speech recognition (ASR) tasks. However, most existing works focus on building ASR models where train and test data are drawn from the same domain. This results in poor generalization characteristics on mismatched-domains: e.g., end-to-end models trained on short segments perform poorly when evaluated on longer utterances. In this work, we analyze the generalization properties of streaming and non-streaming recurrent neural network transducer (RNN-T) based end-to-end models in order to identify model components that negatively affect generalization performance. We propose two solutions: combining multiple regularization techniques during training, and using dynamic overlapping inference. On a long-form YouTube test set, when the non-streaming RNN-T model is trained with shorter segments of data, the proposed combination improves word error rate (WER) from 22.3% to 14.8%; when the streaming RNN-T model trained on short Search queries, the proposed techniques improve WER on the YouTube set from 67.0% to 25.3%. Finally, when trained on Librispeech, we find that dynamic overlapping inference improves WER on YouTube from 99.8% to 33.0%.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Yu Zhang
3. Tara N. Sainath
4. Yonghui Wu
5. Bo Li
6. Chung-Cheng Chiu
7. Quoc V. Le
8. Dong Yu

In Safety, Robustness & Interpretability

1. Arun Narayanan
2. Khe Chai Sim
3. Bo Li
4. Bhuvana Ramabhadran
5. Zhiyun Lu
6. Quoc V. Le
7. Ananya Misra
8. Franccoise Beaufays

In Speech & Audio

1. Yu Zhang
2. Jinyu Li
3. Tara N. Sainath
4. Yonghui Wu
5. Chung-Cheng Chiu
6. Bo Li
7. Ruoming Pang
8. Wei Han

In Signal Processing

1. Jinyu Li
2. Yu Zhang
3. Tara N. Sainath
4. Yonghui Wu
5. Chung-Cheng Chiu
6. Bo Li
7. Dong Yu
8. Ruoming Pang

Citations in the corpus

• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition (2021, background): “1 of [68] for details on the architecture/training of this model.”
• ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context (2020, methodology): “We can see that ContextNet outperforms the previous best architecture from [35], which is a combination of convolution and bidirectional LSTM, by 12% relatively with fewer parameters and FLOPS.”
• Improving Streaming Automatic Speech Recognition with Non-Streaming Model Distillation on Unsupervised Data (2020, methodology): “The target is a sequence of word piece tokens [19, 18, 20].”
• Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models (2021, methodology): “The second teacher model we are considering is the YT-RNNT model [23].”