Prompttts: Controllable Text-To-Speech With Text Descriptions

By Zhifang Guo, Yichong Leng, Yihan Wu, Sheng Zhao, Xuejiao Tan. 2022

201 citations recorded by the source corpus.

Using a text description as prompt to guide the generation of text or images (e.g., GPT-3 or DALLE-2) has drawn wide attention recently. Beyond text and image generation, in this work, we explore the possibility of utilizing text descriptions to guide speech synthesis. Thus, we develop a text-to-speech (TTS) system (dubbed as PromptTTS) that takes a prompt with both style and content descriptions as input to synthesize the corresponding speech. Specifically, PromptTTS consists of a style encoder and a content encoder to extract the corresponding representations from the prompt, and a speech decoder to synthesize speech according to the extracted style and content representations. Compared with previous works in controllable TTS that require users to have acoustic knowledge to understand style factors such as prosody and pitch, PromptTTS is more user-friendly since text descriptions are a more natural way to express speech style (e.g., "A lady whispers to her friend slowly"). Given that there is no TTS dataset with prompts, to benchmark the task of PromptTTS, we construct and release a dataset containing prompts with style and content information and the corresponding speech. Experiments show that PromptTTS can generate speech with precise style control and high speech quality. Audio samples and our dataset are publicly available1.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chao Weng
2. Dong Yu
3. Dongchao Yang
4. Rongjie Huang
5. Xuankai Chang
6. Andros Tjandra
7. Songxiang Liu
8. Jinyu Li

In Signal Processing

1. Chao Weng
2. Dong Yu
3. Rongjie Huang
4. Dongchao Yang
5. Xuankai Chang
6. Songxiang Liu
7. Jinyu Li
8. Andros Tjandra

Citations in the corpus

• Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “We use the same contrastive loss for model training following Wu et al. (2023); Radford et al. (2021), where τ is a learnable parameter.”
• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, background): “…textual description, original sounds sounds Speech dereverberation (SD) ♣♢ (Wu et al., 2016) reverberant speech speech Instruct TTS (I-TTS) ♢ (Guo et al., 2023) phoneme, textual instruction speech Speech Edit (S-Edit) (Tae et al., 2021) ♢ phoneme (with duration), original speech speech…”
• InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, methodology): “Table I shows example style prompts in our dataset, and we also compare NLSpeech with other existing related corpora, including the FSNR0 corpus [31] and the PromptSpeech corpus [32].”
• HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “Meanwhile, there are text prompt-based style generation models which can describe a speaking or voice style from text descriptions [61], [62], [63].”