Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models

By Thibault Doutre, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Olivier Siohan, Liangliang Cao. 2021

6 citations recorded by the source corpus.

Streaming end-to-end automatic speech recognition (ASR) systems are widely used in everyday applications that require transcribing speech to text in real-time. Their minimal latency makes them suitable for such tasks. Unlike their non-streaming counterparts, streaming models are constrained to be causal with no future context and suffer from higher word error rates (WER). To improve streaming models, a recent study [1] proposed to distill a non-streaming teacher model on unsupervised utterances, and then train a streaming student using the teachers' predictions. However, the performance gap between teacher and student WERs remains high. In this paper, we aim to close this gap by using a diversified set of non-streaming teacher models and combining them using Recognizer Output Voting Error Reduction (ROVER). In particular, we show that, despite being weaker than RNN-T models, CTC models are remarkable teachers. Further, by fusing RNN-T and CTC models together, we build the strongest teachers. The resulting student models drastically improve upon streaming models of previous work [1]: the WER decreases by 41% on Spanish, 27% on Portuguese, and 13% on French.

Subfields: Artificial Intelligence, Computer Networks and Communications, Efficiency & Compression, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Yu Zhang
3. Tara N. Sainath
4. Yonghui Wu
5. Quoc V. Le
6. Chung-Cheng Chiu
7. Bo Li
8. Dong Yu

In Computer Networks and Communications

1. Bo Li
2. Quoc V. Le
3. Ruoming Pang
4. Aren Jansen
5. Yongqiang Wang

In Efficiency & Compression

1. Quoc V. Le
2. Jiahui Yu
3. Ruoming Pang
4. Tara N. Sainath
5. Chung-Cheng Chiu
6. Yonghui Wu
7. Yu Zhang
8. Wei Han

In Speech & Audio

1. Yu Zhang
2. Jinyu Li
3. Tara N. Sainath
4. Yonghui Wu
5. Chung-Cheng Chiu
6. Ruoming Pang
7. Bo Li
8. Wei Han

Citations in the corpus

• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition (2021, methodology): “Methods for improving the performance of streaming ASR models using models with future context have been studied in [44]–[52].”