Developing Real-Time Streaming Transformer Transducer for Speech Recognition on Large-Scale Dataset

By Xie Chen, Yu Wu, Zhenghao Wang, Shujie Liu, Jinyu Li. 2020

210 citations recorded by the source corpus.

Recently, Transformer based end-to-end models have achieved great success in many areas including speech recognition. However, compared to LSTM models, the heavy computational cost of the Transformer during inference is a key issue to prevent their applications. In this work, we explored the potential of Transformer Transducer (T-T) models for the fist pass decoding with low latency and fast speed on a large-scale dataset. We combine the idea of Transformer- XL and chunk-wise streaming processing to design a streamable Transformer Transducer model. We demonstrate that T-T outperforms the hybrid model, RNN Transducer (RNN-T), and streamable Transformer attention-based encoder-decoder model in the streaming scenario. Furthermore, the runtime cost and latency can be optimized with a relatively small look-ahead.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Yifan Gong
3. Vadim Mazalov
4. Dong Yu
5. Xipeng Qiu
6. Guoli Ye
7. Tianyang Lin
8. Yuxin Wang

In Signal Processing

1. Jinyu Li
2. Yifan Gong
3. Vadim Mazalov
4. Guoli Ye
5. Dong Yu
6. Xipeng Qiu
7. Li Deng
8. Zhong Meng

Citations in the corpus

• VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation (2023, neutral): “The primary end-to-end model structures used for ASR (Li, 2022; Prab-havalkar et al., 2023) are connectionist temporal classification (CTC) (Graves et al., 2006), attention-based encoder-decoder network (AED) (Chiu et al., 2018), and transducer networks with RNN or Trans-former (Chen et al., 2021a).”
• Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “By replacing the LSTM modules with Transformer in the encoder, the RNN-T model becomes Transformer Transducer [85, 88– 90] which has better accuracy than RNN-T due to the modeling power of Transformer.”
• Have Best of Both Worlds: Two-Pass Hybrid and E2E Cascading Framework for Speech Recognition (2021, methodology): “On these two test sets, we test our in-house streaming transformer transducer (T-T) model [7] trained on the same 65k data by either depending on the end-of-speech (EOS) token to do segmentation and recognition, or just forcing it to decode through the whole pre-segmented audio by hybrid system with”
• A Survey of Transformers (2021, methodology): “Audio Speech Transformer[31], Streaming Transformer[15], Reformer-TTS[57], Music Transformer[56]”