A Better and Faster end-to-end Model for Streaming ASR

By Bo Li, Anmol Gulati, Jiahui Yu, Tara N. Sainath, Chung-Cheng Chiu, Arun Narayanan, Shuo-yiin Chang, Ruoming Pang, Yanzhang He, James Qin and 5 more. 2020

133 citations recorded by the source corpus.

End-to-end (E2E) models have shown to outperform state-of-the-art conventional models for streaming speech recognition [1] across many dimensions, including quality (as measured by word error rate (WER)) and endpointer latency [2]. However, the model still tends to delay the predictions towards the end and thus has much higher partial latency compared to a conventional ASR model. To address this issue, we look at encouraging the E2E model to emit words early, through an algorithm called FastEmit [3]. Naturally, improving on latency results in a quality degradation. To address this, we explore replacing the LSTM layers in the encoder of our E2E model with Conformer layers [4], which has shown good improvements for ASR. Secondly, we also explore running a 2nd-pass beam search to improve quality. In order to ensure the 2nd-pass completes quickly, we explore non-causal Conformer layers that feed into the same 1st-pass RNN-T decoder, an algorithm called Cascaded Encoders [5]. Overall, the Conformer RNN-T with Cascaded Encoders offers a better quality and latency tradeoff for streaming ASR.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Multitask Training with Text Data for End-to-End Speech Recognition (2020, background): “Various model architectures [6, 7, 8, 9, 10] and training schemes [11, 12, 13] were proposed.”
End-to-End Speech Recognition: A Survey (2023, background): “In 2020, for the Pixel 5 smartphone [333], the system was improved further to reduce user-perceived latency (i.e., the time between when the user speaks, and when words appear on the device).”
Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages (2023, background): “9× slower than the 100M-parameter streaming model [87], primarily due to the fact that our models operate at batch processing mode.”
Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “Note that the second-pass model brings additional latency and computational costs to ASR systems, therefore, careful designs have been studied to hide these costs in commercial systems [26, 145, 236].”