CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network
By Vincent Wan, Chun-an Chan, Tom Kenter, Jakub Vít, R. Clark. 2019
79 citations recorded by the source corpus.
The prosodic aspects of speech signals produced by current text-to-speech systems are typically averaged over training material, and as such lack the variety and liveliness found in natural speech. To avoid monotony and averaged prosody contours, it is desirable to have a way of modeling the variation in the prosodic aspects of speech, so audio signals can be synthesized in multiple ways for a given text. We present a new, hierarchically structured conditional variational autoencoder to generate prosodic features (fundamental frequency, energy and duration) suitable for use with a vocoder or a generative model like WaveNet. At inference time, an embedding representing the prosody of a sentence may be sampled from the variational layer to allow for prosodic variation. To efficiently capture the hierarchical nature of the linguistic input (words, syllables and phones), both the encoder and decoder parts of the auto-encoder are hierarchical, in line with the linguistic structure, with layers being clocked dynamically at the respective rates. We show in our experiments that our dynamic hierarchical network outperforms a non-hierarchical state-of-the-art baseline, and, additionally, that prosody transfer across sentences is possible by employing the prosody embedding of one sentence to generate the speech signal of another.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- A Survey on Neural Speech Synthesis (2021, methodology): “[155] predict prosody features from frame and phoneme levels to syllable level, and concatenate with word- and sentencelevel features.”
- Improving the Prosody of RNN-Based English Text-To-Speech Synthesis by Incorporating a BERT Model (2020, methodology): “It is fully described in [3] and not discussed further here for brevity.”
- Hierarchical Prosody Modeling for Non-Autoregressive Speech Synthesis (2020, methodology): “[13, 14] used an autoregressive model to predict frame-level prosody values and mel-spectrogram from hierarchical linguistic features.”
- Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling (2020, methodology): “Early neural parametric synthesis models (Zen et al., 2013) require explicit alignments between input and target and include durations as part of the bag of features used to generate vocoder parameters, and they continue to be used with the advent of the end-to-end neural vocoder WaveNet (Oord et al”
- Semi-Supervised Generative Modeling for Controllable Speech Synthesis (2019, background): “More recently, attempts have also been made to combine probabilistic latent variable models trained using SGVB (Akuzawa et al., 2018; Wan et al., 2019).”
- Prosody Transfer in Neural Text to Speech Using Global Pitch and Loudness Features (2019, background): “[14] proposed a hierarchical conditional VAE which utilizes both prosodic features such as pitch, loudness and phoneme duration, as well as linguistic features to learn a prosody embedding.”
- Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “CHiVE [96] is a conditional VAE model with a hierarchical structure.”
- Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation (2020, background): “The notable results are shown not only for single speaker TTS modeling [3, 4, 5, 6, 7], multi-speaker TTS modeling [8, 9] but also for cloning prosody style [10, 11, 12, 13, 14] and speaker characteristics [15, 16, 17, 18].”
- Unsupervised Speech Decomposition via Triple Information Bottleneck (2020, background): “CHiVE (Kenter et al., 2019) introduces a hierarchical encoder-decoder structure that is conditioned on a set of prosodic features and linguistic features.”