Rnn-Transducer with Stateless Prediction Network

By Mohammadreza Ghodsi, Xiaofeng Liu, James Apfel, Rodrigo Cabrera, Eugene Weinstein. 2020

139 citations recorded by the source corpus.

The RNN-Transducer (RNNT) outperforms classic Automatic Speech Recognition (ASR) systems when a large amount of supervised training data is available. For low-resource languages, the RNNT models overfit, and can not directly take advantage of additional large text corpora as in classic ASR systems.We focus on the prediction network of the RNNT, since it is believed to be analogous to the Language Model (LM) in the classic ASR systems. We pre-train the prediction network with text-only data, which is not helpful. Moreover, removing the recurrent layers from the prediction network, which makes the prediction network stateless, performs virtually as well as the original RNNT model, when using wordpieces. The stateless prediction network does not depend on the previous output symbols, except the last one. Therefore it simplifies the RNNT architectures and the inference.Our results suggest that the RNNT prediction network does not function as the LM in classical ASR. Instead, it merely helps the model align to the input audio, while the RNNT encoder and joint networks capture both the acoustic and the linguistic information.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Libriheavy: A 50,000 Hours ASR Corpus with Punctuation Casing and Context (2023, background): “Different from the setting in [26] which only has an embedding layer, an extra 1-”
Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS (2023, methodology): “The label decoder employs a stateless decoder [31], which consists of an embedding layer followed by a 512-dim Conv1D layer.”
End-to-End Speech Recognition: A Survey (2023, background): “On the decoder side, research for transducer models has shown that a large LSTM decoder can be replaced with a simple embedding lookup table, that attends to only a few previous tokens from the model [47], [104], [105], [106], [107].”
Fast and Parallel Decoding for Transducer (2022, methodology): “In the stateless transducer as in [8] the decoder network requires only a finite context, for instance two symbols.”
Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “[68], the prediction network in RNNT does not fully function as an LM.”
Less is More: Improved RNN-T Decoding Using Limited Label Context and Path Merging (2020, background): “We also note that our results are in contrast to previous studies with have been conducted on smaller-scale tasks with limited training data [6, 16].”