Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron

By R. Skerry-Ryan, Eric Battenberg, Ying Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron J. Weiss, R. Clark, Rif A. Saurous. 2018

611 citations recorded by the source corpus.

We present an extension to the Tacotron speech synthesis architecture that learns a latent embedding space of prosody, derived from a reference acoustic representation containing the desired prosody. We show that conditioning Tacotron on this learned embedding space results in synthesized audio that matches the prosody of the reference signal with fine time detail even when the reference and synthesis speakers are different. Additionally, we show that a reference prosody embedding can be used to synthesize text that is different from that of the reference utterance. We define several quantitative and subjective metrics for evaluating prosody transfer, and report results with accompanying audio samples from single-speaker and 44-speaker Tacotron models on a prosody transfer task.

Subfields: Artificial Intelligence, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Speech & Audio

In Signal Processing

Citations in the corpus

Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis (2018, background): “A similar spectrogram encoder network, trained without a triplet loss, was shown to work for transferring target prosody to synthesized speech [16].”
BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “To (partially) disentangle speaker information from the speech representations, we introduce a global reference encoder [42].”
AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios (2022, background): “The speaker encoder [2, 22] is a 6-layer convolution network,”
A Survey on Neural Speech Synthesis (2021, methodology): “Prosody/style/emotion is the key information to improve the expressiveness of speech and the vast majority of works on expressive TTS focus on improving the prosody/style/emotion of speech [303, 377, 315, 82, 353, 318].”
Fine-Grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis (2020, methodology): “To avoid explicit labels, the approach in [8] adopts a reference encoder to extract a latent prosodic representation for style imitation.”
LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech (2019, background): “They are often used for building single-speaker TTS voices [12,13,32–35].”
Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis (2018, methodology): “As mentioned previously, (Skerry-Ryan et al., 2018) introduces the reference embedding used in this work, and shows that it can be used to transfer prosody from a reference signal.”
Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search (2020, background): “different speaking styles or various prosodies (Wang et al., 2018; Skerry-Ryan et al., 2018; Jia et al., 2018).”
Hierarchical Generative Modeling for Controllable Speech Synthesis (2018, methodology): “Skerry-Ryan et al. (2018); Wang et al. (2018) model such latent attributes through conditional auto-encoding: in addition to text and a speaker label, a vector inferred from the target speech is passed to the decoder as input, which aims to capture the residual attributes that are not speciﬁed by…”
Location-Relative Attention Mechanisms for Robust Long-Form Speech Synthesis (2019, background): “Separately, some authors [10] have moved back toward the purely location-based GMM attention introduced by Graves in [1], and some have proposed stabilizing GMM attention by using soft-plus nonlinearities in place of the exponential function [11, 12].”
Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning (2019, background): “It is becoming easier to manipulate existing videos of people: replacing a person’s facial expressions or movements with those of another person [6, 7], and/or overlaying synthesized speech which imitates a person’s voice [8].”
Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis (2020, methodology): “…of the data while not requiring any additional Prenet or Postnet layer (Wang et al., 2017), nor compound loss functions required by most state of the art models like (Shen et al., 2017; Arik et al., 2017b;a; Ping et al., 2017; Skerry-Ryan et al., 2018; Wang et al., 2018; Bi´nkowski et al., 2019).”
InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, methodology): “3) AXY test: We propose to use AXY test [7] to assess the style relevance between the generated speech with its”
Generating Diverse and Natural Text-to-Speech Samples Using a Quantized Fine-Grained VAE and Autoregressive Prosody Prior (2020, background): “Many recent end-to-end TTS models aiming to capture these factors extract a latent representation from the target speech, and factorize the observed attributes, such as speaker and text information, out of the prosody latent space [6, 8, 11, 12].”
Parallel Tacotron: Non-Autoregressive and Controllable TTS (2020, methodology): “As a baseline we used the Tacotron 2 model [6] with some small modifications: specifically it used Gaussian mixture model (GMM) attention [40, 41] and a reduction factor of 2 [2], both of which have been shown to improve the robustness [24].”
MultiSpeech: Multi-Speaker Text to Speech with Transformer (2020, background): “Several works have built multi-speaker text to speech systems based on RNN [13, 14] and CNN [6, 7].”