Using Vaes and Normalizing Flows for One-Shot Text-To-Speech Synthesis of Expressive Speech

By Vatsal Aggarwal, Marius Cotescu, Nishant Prateek, Jaime Lorenzo-Trueba, Roberto Barra-Chicote. 2019

33 citations recorded by the source corpus.

We propose a Text-to-Speech method to create an unseen expressive style using one utterance of expressive speech of around one second. Specifically, we enhance the disentanglement capabilities of a state-of-the-art sequence-to-sequence based system with a Variational AutoEncoder (VAE) and a Householder Flow. The proposed system provides a 22% KL-divergence reduction while jointly improving perceptual metrics over state-of-the-art. At synthesis time we use one example of expressive style as a reference input to the encoder for generating any text in the desired style. Perceptual MUSHRA evaluations show that we can create a voice with a 9% relative naturalness improvement over standard Neural Text-to-Speech, while also improving the perceived emotional intensity (59 compared to the 55 of neutral speech).

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Xinyu Yang
6. Zhaoxi Mu
7. Yizhuo Dong
8. Sheng Zhao

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, background): “Some works [4, 117, 2, 72] also leverage the VAE framework to better model the variance information for expressive synthesis.”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “[2] used VAE and Householder Flow [211] to improve the reference encoder proposed by Skerry-Ryan et al.”
• Improving Performance of Seen and Unseen Speech Style Transfer in End-to-end Neural TTS (2021, methodology): “Similar to [21], the reference encoder consists of a stack of six 2-D convolutional layers cascaded with one unidirectional 128-unit GRU layer, and the architecture of the IAF is the same as that in [31].”
• An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era (2022, methodology): “A probabilistic realization of AE that is sometimes used for EVC as well is the VAE [119], [122], [157], [166], [167].”
• Distribution Augmentation for Low-Resource Expressive Text-To-Speech (2022, unknown): “Many approaches were recently proposed to improve modelling of expressive speech [15, 16, 17, 18].”
• Improving transfer of expressivity for end-to-end multispeaker text-to-speech synthesis (2021, methodology): “For controlling expressivity, these approaches enhance the Tacotron based TTS system by the addition of advanced deep neural network architecture such as variational autoencoder (VAE) [5], Gaussian mixture VAE [2], Global style token [3], FLOW [8].”