Improving Speech Prosody of Audiobook Text-To-Speech Synthesis with Acoustic and Textual Contexts

By Detai Xin, Sharath Adavanne, Federico Ang, Ashish Kulkarni, Shinnosuke Takamichi, Hiroshi Saruwatari. 2022

20 citations recorded by the source corpus.

We present a multi-speaker Japanese audiobook text-to-speech (TTS) system that leverages multimodal context information of preceding acoustic context and bilateral textual context to improve the prosody of synthetic speech. Previous work either uses unilateral or single-modality context, which does not fully represent the context information. The proposed method uses an acoustic context encoder and a textual context encoder to aggregate context information and feeds it to the TTS model, which enables the model to predict context-dependent prosody. We conducted comprehensive objective and subjective evaluations on a multi-speaker Japanese audiobook dataset. Experimental results demonstrate that the proposed method significantly outperforms two previous works. Additionally, we present insights about the different choices of context - modalities, lateral information and length - for audiobook TTS that have never been discussed in the literature before.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sri Karlapati
2. Alexis Moinet
3. Thomas Drugman
4. Bastian Schnell
5. Louis-philippe Morency

In Signal Processing

1. Alexis Moinet
2. Thomas Drugman
3. Bastian Schnell

Citations in the corpus

• eCat: An End-to-End Model for Multi-Speaker TTS & Many-to-Many Fine-Grained Prosody Transfer (2023, background): “[15] also use acoustic and textual context from surrounding sentences to synthesise speech with contextually appropriate prosody.”