Speech Recognition with Augmented Synthesized Speech

By Andrew Rosenberg, Yu Zhang, Bhuvana Ramabhadran, Ye Jia, Pedro J. Moreno Mengibar, Yonghui Wu, Zelin Wu. 2019

151 citations recorded by the source corpus.

Recent success of the Tacotron speech synthesis architecture and its variants in producing natural sounding multi-speaker synthesized speech has raised the exciting possibility of replacing expensive, manually transcribed, domain-specific, human speech that is used to train speech recognizers. The multi-speaker speech synthesis architecture can learn latent embedding spaces of prosody, speaker and style variations derived from input acoustic representations thereby allowing for manipulation of the synthesized speech. In this paper, we evaluate the feasibility of enhancing speech recognition performance using speech synthesis using two corpora from different domains. We explore algorithms to provide the necessary acoustic and lexical diversity needed for robust speech recognition. Finally, we demonstrate the feasibility of this approach as a data augmentation strategy for domain-transfer. We find that improvements to speech recognition performance is achievable by augmenting training data with synthesized material. However, there remains a substantial gap in performance between recognizers trained on human speech those trained on synthesized speech.

Subfields: Artificial Intelligence, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

Citations in the corpus

You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation (2020, methodology): “[9] used a hierarchical version of variational autoencoder (VAE) [18] to model prosody.”
Semi-Supervision in ASR: Sequential MixMatch and Factorized TTS-Based Augmentation (2021, methodology): “Successful incorporation of unspoken text was demonstrated in a variety of tasks by connecting ASR and TTS via an end-to-end differentiable loss [1, 2, 4, 11].”
Improving Speech Recognition Using Consistent Predictions on Synthesized Speech (2020, background): “[22] and [24] report competitive performance with 4.7 and 4.8 WERs.”
LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition (2020, unknown): “• Inthelow-resourcesetting,thesingle-speakerhigh-qualitypaired dataarereducedtodozensofminutesinTTS[2,12,23,31]while themulti-speakerlow-qualitypaireddataisreducedtodozens ofhoursinASR[16,32,33,39],comparedtothatintherich-resourcesetting.”
Injecting Text in Self-Supervised Speech Pretraining (2021, methodology): “Training methodologies to jointly learn from unpaired speech and text such as Deep Chain [14], cycle-consistency training [15], and augmentation approaches [16, 17] are becoming increasingly popular.”
Data Augmentation for ASR Using TTS Via a Discrete Representation (2021, background): “Several studies have used speaker information to generate multi-speaker speech such as a speaker ID [14], VAE latent variables [15], pre-trained speaker veriﬁcation model [21], and a global style token (GST) [16].”
Machine Speech Chain (2020, methodology): “Rosenberg et al. [32] explored the effect of data augmentation by using TTS on the larger experiment.”
A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks (2020, background): “In order to alleviate the data scarcity issue, different approaches have been studied, including acoustic [11, 12] and linguistic aspects [13, 14].”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “straightforward approach is to simply use multi-speaker TTS to generate the waveform with various acoustic variations [314], [315], [316], [317], [318].”
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021, background): “For example, Hayashi et al. (2018) and Rosenberg et al. (2019) propose to leverage speech synthesis systems to generate paired text and speech data for ASR training; Tjandra et al. (2017), Hori et al. (2019), and Baskar et al. (2019) chain ASR and TTS together to form a loop for semi-supervised…”