Semi-Supervised Generative Modeling for Controllable Speech Synthesis

By Raza Habib, Soroosh Mariooryad, Matt Shannon, Eric Battenberg, R. Skerry-Ryan, Daisy Stanton, David Kao, Tom Bagby. 2019

48 citations recorded by the source corpus.

We present a novel generative model that combines state-of-the-art neural text-to-speech (TTS) with semi-supervised probabilistic latent variable models. By providing partial supervision to some of the latent variables, we are able to force them to take on consistent and interpretable purposes, which previously hasn't been possible with purely unsupervised TTS models. We demonstrate that our model is able to reliably discover and control important but rarely labelled attributes of speech, such as affect and speaking rate, with as little as 1% (30 minutes) supervision. Even at such low supervision levels we do not observe a degradation of synthesis quality compared to a state-of-the-art baseline. Audio samples are available on the web.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Wei-Ning Hsu
5. Frank K. Soong
6. David F. Harwath
7. James R. Glass
8. Chenpeng Du

In Signal Processing

1. Wei-Ning Hsu
2. Xu Tan
3. Tao Qin
4. Tie-Yan Liu
5. David F. Harwath
6. James R. Glass
7. Frank K. Soong
8. Yiwei Guo

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “Technique Description Work Disentangling with Adversarial Training Disentanglement for control [218, 117, 275, 427] Cycle Consistency/Feedback for Control Enhance style/timbre generation [198, 380, 202, 30, 191] Semi-Supervised Learning for Control Use VAE and adversarial training [100, 116, 117, 42”
• Controllable Neural Prosody Synthesis (2020, background): “Recently, significant effort has gone into disentangling the latent spaces of high-quality neural vocoders to recover explicit prosodic control [16, 17, 18, 19].”
• Generative Pre-training for Speech with Flow Matching (2023, background): “For example, generative TTS models (Habib et al., 2019) allow different emotions to be sampled given a fixed text as discriminative models produce a fixed output.”
• Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, methodology): “[30] Raza Habib, Soroosh Mariooryad, Matt Shannon, Eric Battenberg, RJ Skerry-Ryan, Daisy Stanton, David Kao, and Tom Bagby.”
• Unsupervised Word-Level Prosody Tagging for Controllable Speech Synthesis (2022, background): “[4, 5] both provide solutions to control specific acoustic aspects of phone-level speech.”