Low-Resource Expressive Text-To-Speech Using Data Augmentation

By Goeric Huybrechts, Thomas Merritt, Giulia Comini, Bartek Perz, Raahil Shah, Jaime Lorenzo-Trueba. 2020

65 citations recorded by the source corpus.

While recent neural text-to-speech (TTS) systems perform remarkably well, they typically require a substantial amount of recordings from the target speaker reading in the desired speaking style. In this work, we present a novel 3-step methodology to circumvent the costly operation of recording large amounts of target data in order to build expressive style voices with as little as 15 minutes of such recordings. First, we augment data via voice conversion by leveraging recordings in the desired speaking style from other speakers. Next, we use that synthetic data on top of the available recordings to train a TTS model. Finally, we fine-tune that model to further increase quality. Our evaluations show that the proposed changes bring significant improvements over non-augmented models across many perceived aspects of synthesised speech. We demonstrate the proposed approach on 2 styles (news-caster and conversational), on various speakers, and on both single and multi-speaker models, illustrating the robustness of our approach.1

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Xinyu Yang
6. Zhaoxi Mu
7. Yizhuo Dong
8. Sheng Zhao

In Signal Processing

1. Xu Tan
2. Tie-Yan Liu
3. Tao Qin
4. Frank K. Soong
5. Zhaoxi Mu
6. Xinyu Yang
7. Yizhuo Dong
8. Panagiotis Tzirakis

Citations in the corpus

• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “Cars displayed in these showrooms with ENERGY-EFFICIENT AND GREEN decals prominently featured across the windshield aren’t announcing environmentalism; they’re virtue signaling.”
• A Survey on Neural Speech Synthesis (2021, methodology): “Techniques Data Work Self-supervised Training Unpaired text or speech [51, 368, 433, 78, 140, 346, 197, 352, 71] Cross-lingual Transfer Paired text and speech [42, 390, 12, 60, 271, 105] Cross-speaker Transfer Paired text and speech [216, 125, 59, 39] Speech chain/Back transformation Unpaired text o”
• Distribution Augmentation for Low-Resource Expressive Text-To-Speech (2022, unknown): “The authors in [8, 9] applied the CopyCat VC model [14] to generate expressive synthetic speech in low-resource data settings.”
• Improving Emotional TTS with an Emotion Intensity Input from Unsupervised Extraction (2021, methodology): “[7] have used voice conversion to convert expressive recordings to the target speaker.”
• An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era (2022, background): “We further note that several works are investigating explicit prosodic or rhythmic control [24], [25], [26], [27], [28].”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “[81] used a voice conversion (VC) model to convert the voice data of other speakers into the voice of the target speaker for data augmentation, then trained the TTS model with the expanded speech data, and finally used the real voice data of the target speaker for fine-tuning.”