Two-Pass End-to-End Speech Recognition

By Tara N. Sainath, Ruoming Pang, David Rybach, Yanzhang He, Rohit Prabhavalkar, Wei Li, Mirkó Visontai, Qiao Liang, Trevor Strohman, Yonghui Wu and 2 more. 2019

163 citations recorded by the source corpus.

The requirements for many applications of state-of-the-art speech recognition systems include not only low word error rate (WER) but also low latency. Specifically, for many use-cases, the system must be able to decode utterances in a streaming fashion and faster than real-time. Recently, a streaming recurrent neural network transducer (RNN-T) end-to-end (E2E) model has shown to be a good candidate for on-device speech recognition, with improved WER and latency metrics compared to conventional on-device models [1]. However, this model still lags behind a large state-of-the-art conventional model in quality [2]. On the other hand, a non-streaming E2E Listen, Attend and Spell (LAS) model has shown comparable quality to large conventional models [3]. This work aims to bring the quality of an E2E streaming model closer to that of a conventional system by incorporating a LAS network as a second-pass component, while still abiding by latency constraints. Our proposed two-pass model achieves a 17%-22% relative reduction in WER compared to RNN-T alone and increases latency by a small fraction over RNN-T.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Multitask Training with Text Data for End-to-End Speech Recognition (2020, methodology): “It is used in a hybrid network designed for two-pass recognition, a transducer for streaming recognition followed by a non-causal rescoring pass using an attention-based decoder [29].”
Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models (2021, background): “Recent research has focused on improving streaming ASR by trading off latency [11, 3, 12].”
An Attention-Based Joint Acoustic and Text on-Device End-To-End Model (2020, methodology): “While these models have shown competitive performance to conventional models [2, 9], their performance still lags behind on rare words, such as long-tail named entities.”
End-to-End Speech Recognition: A Survey (2023, methodology): “Pang et al. [87] explore combining the beneﬁts of RNN-T and AED.”
Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “In [232], a two-pass model was proposed to use an AED decoder to attend the encoder output of a streaming RNN-T model.”
Recognizing Long-Form Speech Using Streaming End-to-End Models (2019, background): “On medium-scale tasks such as Librispeech [15] or largescale voice-search tasks [16, 17, 18], such E2E models have been shown to outperform conventional ASR systems.”
Universal ASR: Unify and Improve Streaming ASR with Full-context Modeling (2020, methodology): “Sainath et al. further proposed to improve quality by using two-pass models (Sainath et al., 2019), i.e ., a second-pass LAS-based rescore model on top of the hypotheses from ﬁrst-pass RNN-T streaming output.”
Improving Streaming Automatic Speech Recognition with Non-Streaming Model Distillation on Unsupervised Data (2020, background): “In the past few years, many research efforts have been devoted to improving streaming ASR [10, 11, 12].”
Less is More: Improved RNN-T Decoding Using Limited Label Context and Path Merging (2020, background): “The ability to create dense lattices which represent alternative hypotheses has a number of potential applications; future work will investigate its effectiveness in improving word-level confidence estimates [24], and in improving WER by rescoring lattices in the second-pass [36] taking advantage of”
Knowledge Distillation from Offline to Streaming RNN Transducer for End-to-End Speech Recognition (2020, background): “Streaming ASR is one of the important requirements in many speech applications, such as closed captioning for television [20] and natural human-machine interactions [21, 22].”