Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis

By Eric Battenberg, Soroosh Mariooryad, Daisy Stanton, R. Skerry-Ryan, Matt Shannon, David Kao, Tom Bagby. 2019

48 citations recorded by the source corpus.

Recent work has explored sequence-to-sequence latent variable models for expressive speech synthesis (supporting control and transfer of prosody and style), but has not presented a coherent framework for understanding the trade-offs between the competing methods. In this paper, we propose embedding capacity (the amount of information the embedding contains about the data) as a unified method of analyzing the behavior of latent variable models of speech, comparing existing heuristic (non-variational) methods to variational methods that are able to explicitly constrain capacity using an upper bound on representational mutual information. In our proposed model (Capacitron), we show that by adding conditional dependencies to the variational posterior such that it matches the form of the true posterior, the same model can be used for high-precision prosody transfer, text-agnostic style transfer, and generation of natural-sounding prior samples. For multi-speaker models, Capacitron is able to preserve target speaker identity during inter-speaker prosody transfer and when drawing samples from the latent prior. Lastly, we introduce a method for decomposing embedding capacity hierarchically across two sets of latents, allowing a portion of the latent variability to be specified and the remaining variability sampled from a learned prior. Audio examples are available on the web.

Subfields: Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Speech & Audio

Citations in the corpus

You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation (2020, positive): “1) Architecture details: Tacotron encoder and decoder hyperparameters follow [29]: base dimensions are 256 with extensions where concatenation is necessary.”
GAN Vocoder: Multi-Resolution Discriminator Is All You Need (2021, methodology): “Additionally, dynamic time warping is applied to align the MFCC sequences of TTSbased syntheses to their GT pairs (the GT-based syntheses are aligned to their correspondences by default) as in [33].”
Location-Relative Attention Mechanisms for Robust Long-Form Speech Synthesis (2019, methodology): “Separately, some authors [10] have moved back toward the purely location-based GMM attention introduced by Graves in [1], and some have proposed stabilizing GMM attention by using soft-plus nonlinearities in place of the exponential function [11, 12].”
A Unified Sequence-to-Sequence Front-End Model for Mandarin Text-to-Speech Synthesis (2019, methodology): “Different from [18], we replace exponential activation with softplus activation in GMM attention, which is expected to increase the stability substantially during training [19].”
Semi-Supervised Generative Modeling for Controllable Speech Synthesis (2019, background): “This has been shown to be signiﬁcant by Battenberg et al. (2019).”
Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding (2020, unknown): “…studies proposed several specialized embeddings, each with a different set of respon-sibilities to represent diverse speech attributes (e.g., speaking style prosody, and noise) [15, 16, 17, 18, 19, 20] or switched over to a variable-length embedding to maintain temporal information [21, 22, 23].”