Improving Speech Recognition Using Consistent Predictions on Synthesized Speech

By Gary Wang, Andrew Rosenberg, Zhehuai Chen, Yu Zhang, Bhuvana Ramabhadran, Yonghui Wu, Pedro J. Moreno Mengibar. 2020

54 citations recorded by the source corpus.

Speech synthesis has advanced to the point of being close to indistinguishable from human speech. However, efforts to train speech recognition systems on synthesized utterances have not been able to show that synthesized data can be effectively used to augment or replace human speech. In this work, we demonstrate that promoting consistent predictions in response to real and synthesized speech enables significantly improved speech recognition performance. We also find that training on 460 hours of LibriSpeech augmented with 500 hours of transcripts (without audio) performance is within 0.2% WER of a system trained on 960 hours of transcribed audio. This suggests that with this approach, when there is sufficient text available, reliance on transcribed audio can be cut nearly in half.

Subfields: Artificial Intelligence, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

Citations in the corpus

Multitask Training with Text Data for End-to-End Speech Recognition (2020, methodology): “used a loss term to encourage the ASR model to generate consistent outputs on real and synthesized presentations of the same utterance [25].”
Injecting Text in Self-Supervised Speech Pretraining (2021, methodology): “Training methodologies to jointly learn from unpaired speech and text such as Deep Chain [14], cycle-consistency training [15], and augmentation approaches [16, 17] are becoming increasingly popular.”
Semi-Supervision in ASR: Sequential MixMatch and Factorized TTS-Based Augmentation (2021, methodology): “Over the last few years, text-to-speech (TTS) synthesis-based data augmentation has been shown to be a useful way to train end-to-end models using unspoken text [1–4].”
Data Augmentation for ASR Using TTS Via a Discrete Representation (2021, background): “Wang et al. [18, 19] investigated consistency regularization when TTS is incorporated with ASR.”
Exploring Machine Speech Chain for Domain Adaptation and Few-Shot Speaker Adaptation (2021, methodology): “With the help of consistency loss between ASR prediction on real and synthetic speech, the paired training data for ASR can be reduced by half with little degradation in performance [25].”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “TTS without using paired data [302], [303], [304].”