A Comparison of End-to-End Models for Long-Form Speech Recognition

By Chung-Cheng Chiu, Wei Han, Yu Zhang, Ruoming Pang, Sergey Kishchenko, Patrick Nguyen, Arun Narayanan, Hank Liao, Shuyuan Zhang, Anjuli Kannan and 4 more. 2019

92 citations recorded by the source corpus.

End-to-end automatic speech recognition (ASR) models, including both attention-based models and the recurrent neural network transducer (RNN-T), have shown superior performance compared to conventional systems [1], [2]. However, previous studies have focused primarily on short utterances that typically last for just a few seconds or, at most, a few tens of seconds. Whether such architectures are practical on long utterances that last from minutes to hours remains an open question. In this paper, we both investigate and improve the performance of end-to-end models on long-form transcription. We first present an empirical comparison of different end-to-end models on a real world long-form task and demonstrate that the RNN-T model is much more robust than attention-based systems in this regime. We next explore two improvements to attention-based systems that significantly improve its performance: restricting the attention to be monotonic, and applying a novel decoding algorithm that breaks long utterances into shorter overlapping segments. Combining these two improvements, we show that attention-based end-to-end models can be very competitive to RNN-T on long-form speech recognition.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Tara N. Sainath
3. Yu Zhang
4. Dong Yu
5. Shinji Watanabe
6. Yonghui Wu
7. Chung-Cheng Chiu
8. Bo Li

In Evaluation & Benchmarks

1. Yu Zhang
2. Jinyu Li
3. Zhong Meng
4. Wei Han
5. Chung-Cheng Chiu
6. Yonghui Wu
7. Ankur Bapna
8. Bo Li

In Speech & Audio

1. Jinyu Li
2. Tara N. Sainath
3. Yu Zhang
4. Yonghui Wu
5. Shinji Watanabe
6. Chung-Cheng Chiu
7. Bo Li
8. Ruoming Pang

Citations in the corpus

• ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context (2020, methodology): “We can see that ContextNet outperforms the previous best architecture from [33], which is a combination of convolution and bidirectional LSTM, by 12% relatively with fewer parameters and FLOPS.”
• RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions (2020, methodology): “In addition to improving generalization during training, we attempt to improve generalization during decoding with overlapping inference [21].”
• WhisperX: Time-Accurate Speech Transcription of Long-Form Audio (2023, unknown): “Recent works [11] employ heuristic sliding window style approaches that are prone to errors due to overlapping or incomplete audio (e.g. words being cut halfway through).”
• Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “However, AED models cannot perform well on long utterances [54].”
• Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages (2023, background): “Long-form ASR has been studied in [1, 56, 57]; we propose chunk-wise attention as an alternative solution to chunk-based decoding.”
• Universal ASR: Unify and Improve Streaming ASR with Full-context Modeling (2020, background): “…such as monotonic attention (Raffel et al., 2017; Chiu & Raffel, 2017; Fan et al., 2018; Arivazhagan et al., 2019), GMM attention (Graves, 2013; Chiu et al., 2019), triggered attention (TA) (Moritz et al., 2019), Scout Network (Wang et al., 2020), and approaches that segment encoder output…”
• A Better and Faster end-to-end Model for Streaming ASR (2020, background): “We also replaced the full context self-attention with local self-attention which helps generalize to long utterances [22].”
• End-to-End Speech Recognition: A Survey (2023, background): “sensitive to the length of the acoustic sequences [50], which requires special processing for long-form audio [50].”