Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss

By Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, Shankar Kumar. 2020

539 citations recorded by the source corpus.

In this paper we present an end-to-end speech recognition model with Transformer encoders that can be used in a streaming speech recognition system. Transformer computation blocks based on self-attention are used to encode both audio and label sequences independently. The activations from both audio and label encoders are combined with a feed-forward layer to compute a probability distribution over the label space for every combination of acoustic frame position and label history. This is similar to the Recurrent Neural Network Transducer (RNN-T) model, which uses RNNs for information encoding instead of Transformer encoders. The model is trained with the RNN-T loss well-suited to streaming decoding. We present results on the LibriSpeech dataset showing that limiting the left context for self-attention in the Transformer layers makes decoding computationally tractable for streaming, with only a slight degradation in accuracy. We also show that the full attention version of our model beats the-state-of-the art accuracy on the LibriSpeech benchmarks. Our results also show that we can bridge the gap between full attention and limited attention versions of our model by attending to a limited number of future frames.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Furu Wei
3. Yu Zhang
4. Quoc V. Le
5. Dong Yu
6. Jianfeng Gao
7. Shinji Watanabe
8. Tara N. Sainath

In Model Architectures

1. Quoc V. Le
2. Jinyu Li
3. Shinji Watanabe
4. Furu Wei
5. Yu Zhang
6. Tara N. Sainath
7. Yonghui Wu
8. Li Dong

In Speech & Audio

1. Jinyu Li
2. Yu Zhang
3. Tara N. Sainath
4. Yonghui Wu
5. Chung-Cheng Chiu
6. Shinji Watanabe
7. Ruoming Pang
8. Ye Jia

In Signal Processing

1. Jinyu Li
2. Yu Zhang
3. Dong Yu
4. Shinji Watanabe
5. Tara N. Sainath
6. Yonghui Wu
7. Chung-Cheng Chiu
8. Ruoming Pang

Citations in the corpus

• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, neutral): “Transducer [60] Transf.”
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (2020, methodology): “Following [8], we re-tune the LM fusion parameters (the LM fusion weight [51] and the non-blank reward [53, 54]) at each generation to maximize dev set performance of the fused model.”
• Improved Noisy Student Training for Automatic Speech Recognition (2020, methodology): “We introduce a coverage penalty term [32] in the fusion score with parameter c for LAS models [33], while we use a constant non-blank reward ρ to shift the logits produced by the LM for RNN-T [34] networks [35, 36].”
• Conformer: Convolution-augmented Transformer for Speech Recognition (2020, methodology): “Table 2 compares the (WER) result of our model on LibriSpeech test-clean/test-other with a few state-of-the-art models include: ContextNet [10], Transformer transducer [7], and QuartzNet [9].”
• ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context (2020, methodology): “This is a big improvement over previous CNN based architectures such as QuartzNet[4], and it outperforms transformer and LSTM based models [11, 10, 24, 9, 25].”
• Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, background): “Parti is a sequence-to-sequence model based on the Transformer [4], an architecture critical to performance on many tasks, including machine translation [4], speech recognition [22, 23], conversational modeling [24], image captioning [25], and many others.”
• Developing Real-Time Streaming Transformer Transducer for Speech Recognition on Large-Scale Dataset (2020, methodology): “In [32, 21], the authors set the attention mask to allow a specific number of contextual frames in each transformer layer for both history and the future.”
• slimIPL: Language-Model-Free Iterative Pseudo-Labeling (2020, unknown): “FullAttn T-T [78] 30ms letters RNN-T - - 2.”
• Simplified Self-Attention for Transformer-Based end-to-end Speech Recognition (2020, background): “These models can be combined as well to further boost performance [12–14].”
• Scaling End-to-End Models for Large-Scale Multilingual ASR (2021, methodology): “Secondly, we also adopt a Transformer decoder with masked self attention and cross attention to the encoder outputs [13, 34].”
• Fast, Simpler and More Accurate Hybrid ASR Systems Using Wordpieces (2020, background): “Most recently, neural transducers [24] have shown great potential for both on-device [25] and server [26] use cases.”
• Foundation Transformers (2022, methodology): “Since the transducer framework is proven to obtain better accuracy with low latency, we choose the Transformer Transducer (T-T; Zhang et al. 2020b) as the backbone framework, where the encoder is either Pre-LN Transformer or M AGNETO , and the predictor network is a two-layer LSTM network.”
• GLiT: Neural Architecture Search for Global and Local Image Transformer (2021, methodology): “1D convolution has been used in NLP tasks [24, 8] to model local information.”
• Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition (2021, unknown): “On the contrary, the T-T network is suitable for the streaming task [28].”
• End-to-End Speech Recognition: A Survey (2023, methodology): “The use of self-attention to replace the sequential recurrence in LSTMs was explored in Transformers for ASR [98], [99], [100].”