Expressive TTS Training With Frame and Style Reconstruction Loss
By Rui Liu, Berrak Sisman, Guanglai Gao, Haizhou Li. 2020
85 citations recorded by the source corpus.
We propose a novel training strategy for Tacotron-based text-to-speech (TTS) system that improves the speech styling at utterance level. One of the key challenges in prosody modeling is the lack of reference that makes explicit modeling difficult. The proposed technique doesn’t require prosody annotations from training data. It doesn’t attempt to model prosody explicitly either, but rather encodes the association between input text and its prosody styles using a Tacotron-based TTS framework. This study marks a departure from the style token paradigm where prosody is explicitly modeled by a bank of prosody embeddings. It adopts a combination of two objective functions: 1) frame level reconstruction loss, that is calculated between the synthesized and target spectral features; 2) utterance level style reconstruction loss, that is calculated between the deep style features of synthesized and target speech. The style reconstruction loss is formulated as a perceptual loss to ensure that utterance level speech style is taken into consideration during training. Experiments show that the proposed training strategy achieves remarkable performance and outperforms the state-of-the-art baseline in both naturalness and expressiveness. To our best knowledge, this is the first study to incorporate utterance level perceptual quality as a loss function into Tacotron training for improved expressiveness.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
Citations in the corpus
- A Survey on Neural Speech Synthesis (2021, methodology): “Other works [238, 202, 30, 299, 393, 6] leverage the feedback loss to ensure the controllability on style and speaker embeddings, etc.”
- Emotional Voice Conversion: Theory, Databases and ESD (2021, methodology): “…of auto-encoder (Hsu et al., 2016) and its variants (Hsu et al., 2017; Chou et al., 2019; Wu and Lee, 2020) in disentangling the speaker information from the content, thus they are widely used in speaker voice conversion (Sisman et al., 2020) and singing voice conversion (Lu et al., 2020).”
- StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis (2022, unknown): “…concatenating styles in place of AdaIN dramatically decreases the correlations and lowers rated naturalness, confirming our observation that all previous methods that use concatenation to incorporate style information ([1], [9], [16], [17], [19], [20], [69]) are not as effective as AdaIN.”
- An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning (2020, background): “Tacotron [89], [180], [218]–[220] represents one of the successful text-to-speech (TTS) implementations, that has been extended to voice conversion [3], [183], [221].”
- Improving Performance of Seen and Unseen Speech Style Transfer in End-to-end Neural TTS (2021, background): “, speaker identity, emotion and prosody), which are essential for expressive and diverse voice generation, can be also modelled and controlled [8, 9, 10, 11, 12, 13].”
- DiffProsody: Diffusion-Based Latent Prosody Generation for Expressive Speech Synthesis With Prosody Conditional Adversarial Training (2023, methodology): “[24] proposed a method for modeling prosody using style, perception, and frame-level reconstruction loss.”
- An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era (2022, methodology): “Similarly, Wu et al. [128] achieve this with a capsule network [161], while Kreuk et al. [129], Liu et al. [133], and Lei et al. [134] achieve this via frame-level losses.”