An Attention-Based Joint Acoustic and Text on-Device End-To-End Model
By Tara N. Sainath, Ruoming Pang, Ron Weiss, Yanzhang He, Chung-Cheng Chiu, Trevor Strohman. 2020
16 citations recorded by the source corpus.
Recently, we introduced a two-pass on-device end-to-end (E2E) speech recognition model, which runs RNN-T in the first-pass and then rescores/redecodes the result using a noncausal Listen, Attend and Spell (LAS) decoder. This on-device model obtained similar performance to a state-of-the-art conventional model. However, like many E2E models, it suffers from being trained only on supervised audio-text pairs and thus performs poorly on rare words compared to a conventional model which incorporates a language model trained on a much larger text corpus. In this work, we introduce a joint acoustic and text decoder (JATD) into the LAS decoder, which makes it possible to incorporate a much larger text corpus into training. We find that the JATD model obtains in a 3-10% relative improvement in WER compared to a LAS decoder trained only on supervised audio-text pairs across a variety of proper noun test sets.
Subfields: Artificial Intelligence, Multimodal Learning, Model Architectures, Speech & Audio
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Multimodal Learning
In Model Architectures
In Speech & Audio
Citations in the corpus
- Multitask Training with Text Data for End-to-End Speech Recognition (2020, methodology): “The joint acoustic and text decoder (JATD) model [28] incorporates the text transcribed by a conventional ASR model in order to work in two modes, ASR and language modeling.”
- Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “In [177, 233], a joint acoustic and text decoder was proposed to leverage the text data by synthesizing TTS audio from the text to improve the decoder in AED.”