Sequence to Sequence Neural Speech Synthesis with Prosody Modification Capabilities

By Slava Shechtman, Alexander Sorin. 2019

35 citations recorded by the source corpus.

Modern sequence to sequence neural TTS systems provide close to natural speech quality. Such systems usually comprise a network converting linguistic/phonetic features sequence to an acoustic features sequence, cascaded with a neural vocoder. The generated speech prosody (i.e. phoneme durations, pitch and loudness) is implicitly present in the acoustic features, being mixed with spectral information. Although the speech sounds natural, its prosody realization is randomly chosen and cannot be easily altered. The prosody control becomes an even more difficult task if no prosodic labeling is present in the training data. Recently, much progress has been achieved in unsupervised speaking style learning and generation, however human inspection is still required after the training for discovery and interpretation of the speaking styles learned by the system. In this work we introduce a fully automatic method that makes the system aware of the prosody and enables sentence-wise speaking pace and expressiveness control on a continuous scale. While being useful by itself in many applications, the proposed prosody control can also improve the overall quality and expressiveness of the synthesized speech, as demonstrated by subjective listening evaluations. We also propose a novel augmented attention mechanism, that facilitates better pace control sensitivity and faster attention convergence.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

Citations in the corpus

Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning (2019, methodology): “The differential spectral loss [25] was adopted to boost the performance of TTS model.”
Supervised and unsupervised approaches for controlling narrow lexical focus in sequence-to-sequence speech synthesis (2021, methodology): “These have already been described in [5] and are summarized here as follows.”
Unsupervised Speech Decomposition via Triple Information Bottleneck (2020, background): “In speech generation tasks, disentanglement is the foundation of many applications, such as voice conversion (Chou & Lee, 2019), prosody modifcation (Shechtman & Sorin, 2019), emotional speech synthesis (Pell et al., 2011), and low bit-rate speech encoding (Schroeder & Atal, 1985), to name a few.”
Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation (2020, methodology): “The differential spectral loss [26] is also adopted to boost the performance of the TTS model.”
Analysis and Assessment of Controllability of an Expressive Deep Learning-based TTS system (2021, background): “Different supervised approaches were also proposed to control specific characteristics of expressiveness [11, 12].”