Universal ASR: Unify and Improve Streaming ASR with Full-context Modeling

By Jiahui Yu, Wei Han, Anmol Gulati, Chung-Cheng Chiu, Bo Li, Tara N. Sainath, Yonghui Wu, Ruoming Pang. 2020

19 citations recorded by the source corpus.

Streaming automatic speech recognition (ASR) aims to emit each hypothesized word as quickly and accurately as possible, while full-context ASR waits for the completion of a full speech utterance before emitting completed hypotheses. In this work, we propose a unified framework, Universal ASR, to train a single end-to-end ASR model with shared weights for both streaming and full-context speech recognition. We show that the latency and accuracy of streaming ASR significantly benefit from weight sharing and joint training of full-context ASR, especially with inplace knowledge distillation. The Universal ASR framework can be applied to recent state-of-the-art convolution-based and transformer-based ASR networks. We present extensive experiments with two state-of-the-art ASR networks, ContextNet and Conformer, on two datasets, a widely used public dataset LibriSpeech and an internal large-scale dataset MultiDomain. Experiments and ablation studies demonstrate that Universal ASR not only simplifies the workflow of training and deploying streaming and full-context ASR models, but also significantly improves both emission latency and recognition accuracy of streaming ASR. With Universal ASR, we achieve new state-of-the-art streaming ASR results on both LibriSpeech and MultiDomain in terms of accuracy and latency.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Improving Streaming Automatic Speech Recognition with Non-Streaming Model Distillation on Unsupervised Data (2020, methodology): “Then, we use both LibriSpeech and LibriLight to train another student streaming model using the same structure as [29].”
End-to-End Speech Recognition: A Survey (2023, methodology): “To address this, methods which run a 2nd-pass beam search have also been explored, particularly in the context of streaming ASR – e.g. cascaded encoder [89], Y-architecture [90] and Universal ASR [91].”
WeNet: Production Oriented Streaming and Non-Streaming End-to-End Speech Recognition Toolkit (2021, background): “Unifying streaming and non-streaming in a single model can reduce the development effort, training cost as well as deployment cost, which is also preferred for production adoption [19, 20, 21, 22].”