Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis
By Gustav Eje Henter, Xin Eric Wang, Junichi Yamagishi. 2018
61 citations recorded by the source corpus.
Generating versatile and appropriate synthetic speech requires control over the output expression separate from the spoken text. Important non-textual speech variation is seldom annotated, in which case output control must be learned in an unsupervised fashion. In this paper, we perform an in-depth study of methods for unsupervised learning of control in statistical speech synthesis. For example, we show that popular unsupervised training heuristics can be interpreted as variational inference in certain autoencoder models. We additionally connect these models to VQ-VAEs, another, recently-proposed class of deep variational autoencoders, which we show can be derived from a very similar mathematical argument. The implications of these new probabilistic interpretations are discussed. We illustrate the utility of the various approaches with an application to acoustic modelling for emotional speech synthesis, where the unsupervised methods for learning expression control (without access to emotional labels) are found to give results that in many aspects match or surpass the previous best supervised approach.
Subfields: Artificial Intelligence, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Signal Processing
Citations in the corpus
- Hierarchical Generative Modeling for Controllable Speech Synthesis (2018, background): “Skerry-Ryan et al. (2018); Wang et al. (2018) model such latent attributes through conditional auto-encoding: in addition to text and a speaker label, a vector inferred from the target speech is passed to the decoder as input, which aims to capture the residual attributes that are not specified by…”
- Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, methodology): “the speaker characteristics and other acoustic properties to be independently controllable from the I2U system (Wang et al., 2018; Hsu et al., 2019; Henter et al., 2018; Akuzawa et al., 2018).”
- Fully-Hierarchical Fine-Grained Prosody Modeling For Interpretable Speech Synthesis (2020, background): “Abundant research has been performed on learning latent representations for styles and prosody [15, 16] such as the use of an utterancelevel VAE in [17].”
- In Other News: a Bi-style Text-to-speech Model for Synthesizing Newscaster Voice with Limited Data (2019, background): “Several works have explored the “controllabi-lity” of style in synthesised speech through latent-variable modelling techniques (Akuzawa et al., 2018; Henter et al., 2018; Hsu et al., 2018).”
- Improved Prosody from Learned F0 Codebook Representations for VQ-VAE Speech Waveform Reconstruction (2020, methodology): “In this paper, we focus on Vector Quantized Variational AutoEncoder (VQ-VAE) [4] since it is a promising selfsupervised representation learning approach suitable for tasks such as voice conversion (VC) [4, 5], text-to-speech (TTS) synthesis [6, 7, 8], speech coding [9], and even music generation [10”
- Improving Prosody Modelling with Cross-Utterance Bert Embeddings for End-to-End Speech Synthesis (2020, methodology): “Alternatively, variational autoencoders [18] can be used to learn Thanks to Xixin Wu, Guangzhi Sun, and Jing Huang for discussions. style or prosody embeddings to achieve a fine-grained control of the synthesized speech [19–24].”
- Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis (2019, methodology): “Existing heuristic (non-variational) end-to-end approaches to prosody and style transfer (Skerry-Ryan et al., 2018; Wang et al., 2018; Lee & Kim, 2019; Henter et al., 2018) typically start with the teacherforced reconstruction loss, (1), used to train Tacotron-like sequence-to-sequence models and si”
- Hierarchical Multi-Grained Generative Model for Expressive Speech Synthesis (2020, methodology): “A VAE is applied to a TTS synthesis system by regarding X and Y as the sequence of acoustic features and the linguistic features, respectively [12, 13, 14].”
- Disentangling Correlated Speaker and Noise for Speech Synthesis via Data Augmentation and Adversarial Factorization (2019, methodology): “Recent development of neural end-to-end TTS models [28, 2] enables control of both labelled and 17 unlabelled speech attributes by conditioning synthesis on both text and learned attribute represen18 tations [29, 23, 12, 1, 7, 11].”
- A Vector Quantized Variational Autoencoder (VQ-VAE) Autoregressive Neural $F_0$ Model for Statistical Parametric Speech Synthesis (2020, background): “In this case e only encodeso’s variation that cannot be explained by x [36].”
- CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network (2019, methodology): “For example (Akuzawa et al., 2018) propose VAE-Loop where the latent space is used to model expression of speech, and (Henter et al., 2018) use a number of VAE related techniques to show that categories of acted emotion can be modeled in an unsupervised fashion.”