AdaSpeech 3: Adaptive Text to Speech for Spontaneous Style

By Yuzi Yan, Xu Tan, Bohan Li, Guangyan Zhang, Tao Qin, Sheng Zhao, Yuan Shen, Weiqiang Zhang, Tie-Yan Liu. 2021

25 citations recorded by the source corpus.

While recent text to speech (TTS) models perform very well in synthesizing reading-style (e.g., audiobook) speech, it is still challenging to synthesize spontaneous-style speech (e.g., podcast or conversation), mainly because of two reasons: 1) the lack of training data for spontaneous speech; 2) the difficulty in modeling the filled pauses (um and uh) and diverse rhythms in spontaneous speech. In this paper, we develop AdaSpeech 3, an adaptive TTS system that fine-tunes a well-trained reading-style TTS model for spontaneous-style speech. Specifically, 1) to insert filled pauses (FP) in the text sequence appropriately, we introduce an FP predictor to the TTS model; 2) to model the varying rhythms, we introduce a duration predictor based on mixture of experts (MoE), which contains three experts responsible for the generation of fast, medium and slow speech respectively, and fine-tune it as well as the pitch predictor for rhythm adaptation; 3) to adapt to other speaker timbre, we fine-tune some parameters in the decoder with few speech data. To address the challenge of lack of training data, we mine a spontaneous speech dataset to support our research this work and facilitate future research on spontaneous TTS. Experiments show that AdaSpeech 3 synthesizes speech with natural FP and rhythms in spontaneous styles, and achieves much better MOS and SMOS scores than previous adaptive TTS systems.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Sheng Zhao
5. Lei He
6. Frank K. Soong
7. Yanqing Liu
8. Bohan Li

Citations in the corpus

• AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios (2022, methodology): “When synthesizing speech for new speakers, few-shot adaptive TTS [10, 11, 12, 13] is usually adopted by first training a source TTS model on a large multi-speaker dataset and then fine-tuning this model on a few speech data of target speakers.”
• A Survey on Neural Speech Synthesis (2021, methodology): “06 AdaSpeech 3 [398] AM ph FF −→melS FF −→wav IS21 2021.”
• RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertion (2022, methodology): “In the first line, methods [7, 8, 9, 10] achieve satisfactory voice quality and similarity by finetuning the pre-trained model on the reference voice before inference.”
• A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech (2023, background): “There is also research (Yan et al. 2021) focusing on modeling characteristics of spontaneous speech, such as filled pauses (i.e., um , uh ).”