Multitask Training with Text Data for End-to-End Speech Recognition

By Peidong Wang, Tara N. Sainath, Ron J. Weiss. 2020

32 citations recorded by the source corpus.

We propose a multitask training method for attention-based end-to-end speech recognition models to better incorporate language level information. We regularize the decoder in a sequence-to-sequence architecture by multitask training it on both the speech recognition task and a next-token prediction language modeling task. Trained on either the 100 hour subset of LibriSpeech or the full 960 hour dataset, the proposed method leads to an 11% relative performance improvement over the baseline and is comparable to language model shallow fusion, without requiring an additional neural network during decoding. Analyses of sample output sentences and the word error rate on rare words demonstrate that the proposed method can incorporate language level information effectively.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Speech & Audio

Citations in the corpus

Robust Speech Recognition via Large-Scale Weak Supervision (2022, methodology): “We attribute this to the 68,000 hours of X → en translation data for these languages in our pre-training dataset which, although noisy, is vastly larger than the 861 hours of training data for X → en translation in CoVoST2.”
Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “This was simplified in [292] by using multitask training with text only data.”