Robust and Fine-grained Prosody Control of End-to-end Speech Synthesis

By Younggun Lee, Taesu Kim. 2018

156 citations recorded by the source corpus.

We propose prosody embeddings for emotional and expressive speech synthesis networks. The proposed methods introduce temporal structures in the embedding networks, thus enabling fine-grained control of the speaking style of the synthesized speech. The temporal structures can be designed either on the speech side or the text side, leading to different control resolutions in time. The prosody embedding networks are plugged into end-to-end speech synthesis networks and trained without any other supervision except for the target speech for synthesizing. It is demonstrated that the prosody embedding networks learned to extract prosodic features. By adjusting the learned prosody features, we could change the pitch and amplitude of the synthesized speech both at the frame level and the phoneme level. We also introduce the temporal normalization of prosody embeddings, which shows better robustness against speaker perturbations during prosody transfer tasks.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A Survey on Neural Speech Synthesis (2021, background): “6) Frame level [184, 155, 48, 427], the most fine-grained information.”
Direct speech-to-speech translation with a sequence-to-sequence model (2019, background): “Other future work includes utilizing weakly supervision to scale up training with synthetic data [21] or multitask learning [19,20], and transferring prosody and other acoustic factors from the source speech to the translated speech following [45–47].”
Location-Relative Attention Mechanisms for Robust Long-Form Speech Synthesis (2019, background): “Attention normalization isn’t a signiﬁcant problem in ﬁne-grained output-to-text alignment, but becomes more of an issue for coarser-grained alignment tasks where the attention window needs to gradually move to the next index (for example in variable-length prosody transfer applications [13]).”
Generating Diverse and Natural Text-to-Speech Samples Using a Quantized Fine-Grained VAE and Autoregressive Prosody Prior (2020, methodology): “The fine-grained VAE structure used to model the prosody at phoneme-level, similar to that in [13], is shown in Fig.”
Controllable Neural Prosody Synthesis (2020, background): “Recently, significant effort has gone into disentangling the latent spaces of high-quality neural vocoders to recover explicit prosodic control [16, 17, 18, 19].”
Teacher-Student Training For Robust Tacotron-Based TTS (2019, methodology): “In these techniques, the key idea is to integrate the conventional TTS pipeline into a unified network and learn the mapping directly from the <text, wav> pair [7–10].”
Emotional Speech Synthesis with Rich and Granularized Control (2019, background): “, Tacotron), the expressiveness of synthesized speech is controlled by conditioning additional embedding vectors that implicitly provide prosodyrelated latent features typically generated by another neural network [8, 9, 10, 11, 12, 13].”
Modeling Prosodic Phrasing With Multi-Task Learning in Tacotron-Based TTS (2020, background): “, pt[5]], t ∈ [1, T ], represents the probability of the phrase break label k.”
Speaking Speed Control of End-to-End Speech Synthesis using Sentence-Level Conditioning (2020, background): “In [5–8], various speech attributes, such as pitch, prosody, and speaking speed are extracted from a reference speech and the TTS system generates a similar style of speech to the reference speech.”
Improving Emotional TTS with an Emotion Intensity Input from Unsupervised Extraction (2021, background): “The lack of these annotations spawned a range of recent works focusing on increased model generalisability by utilising unsupervised methods to extract style embeddings from reference audio on a global [1, 2], clustered [3], or frame level [4, 5].”
HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “For acoustic and semantic representations, we use a low-resolution representation of 50 Hz for e ﬃ cient training.”
Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis (2019, background): “Lee & Kim (2019) partially address the pitch range problem by centering the learned embeddings using speaker-wise means.”
Unsupervised Word-Level Prosody Tagging for Controllable Speech Synthesis (2022, background): “[4] introduces temporal structures in the embedding networks that can control pitch and amplitude either on speech side or text side.”
Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling (2020, methodology): “We instead utilize a ﬁne-grained VAE (FVAE) similar to Sun et al. (2020); Lee & Kim (2019) to model the alignment between the input tokens and the target mel-spectrogram frames, and extract per-token latent features from the target mel-spectrogram based on this alignment.”
Tacotron-Based Acoustic Model Using Phoneme Alignment for Practical Neural Text-to-Speech Systems (2019, methodology): “• Many seq2seq methods introduce phoneme input [14–16, 19, 23–35] rather than character input [10–22].”
Rich Prosody Diversity Modelling with Phone-Level Mixture Density Network (2021, background): “[9] extracts prosody information from acoustic features and uses an attention module to align them with each phoneme encoding.”