InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt

By Dongchao Yang, Songxiang Liu, Rongjie Huang, Guangzhi Lei, Chao Weng, Helen Mei-Ling Meng, Dong Yu. 2023

174 citations recorded by the source corpus.

Expressive text-to-speech (TTS) aims to synthesize speech with varying speaking styles to better reflect human speech patterns. In this study, we attempt to use natural language as a style prompt to control the styles in the synthetic speech, e.g., “Sigh tone in full of sad mood with some helpless feeling”. Considering that there is no existing TTS corpus that is suitable to benchmark this novel task, we first construct a speech corpus whose speech samples are annotated with not only content transcriptions but also style descriptions in natural language. Then we propose an expressive TTS model, named InstructTTS, which is novel in the sense of the following aspects: (1) We fully take advantage of self-supervised learning and cross-modal metric learning and propose a novel three-stage training procedure to obtain a robust sentence embedding model that can effectively capture semantic information from the style prompts and control the speaking style in the generated speech. (2) We propose to model acoustic features in discrete latent space and train a novel discrete diffusion probabilistic model to generate vector-quantized (VQ) acoustic tokens rather than the commonly-used mel spectrogram. (3) We jointly apply mutual information (MI) estimation and minimization during acoustic model training to minimize style-speaker and style-content MI, avoiding possible content and speaker information leakage from the style prompt. Extensive objective and subjective evaluation has been conducted to verify the effectiveness and expressiveness of InstructTTS. Experimental results show that InstructTTS can synthesize high-fidelity and natural speech with style prompts controlling the speaking style.

Subfields: Artificial Intelligence, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Zhou Zhao
3. Jinyu Li
4. Yi Ren
5. Dong Yu
6. Tao Qin
7. Rongjie Huang
8. Jinglin Liu

In Speech & Audio

1. Jinyu Li
2. Xu Tan
3. Yi Ren
4. Zhou Zhao
5. Rongjie Huang
6. Jinglin Liu
7. Sheng Zhao
8. Dong Yu

Citations in the corpus

• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “…these systems into four groups based on data representation and modelling methods: 1) Discrete Tokens + Autoregressive [6, 19, 20]; 2) Discrete Tokens + Non-2 autoregressive [13, 21, 22]; 3) Continuous Vectors + Autoregressive [23]; 4) Continuous Vectors + Non-autoregressive [5, 11, 24, 25].”
• Audiobox: Unified Audio Generation with Natural Language Prompts (2023, background): “In objective evaluation, previous speech-oriented studies Guo et al. (2023); Leng et al. (2023); Yang et al. (2023a) often adopt ad-hoc evaluation metrics (e.g., accuracy of pre-defined attributes), making it challenging to generalize to free-form instructions.”
• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, background): “Text (continuous): Text can act as the carrier of human instructions in audio generation tasks (Yang et al., 2023a; Copet et al., 2023).”
• HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec (2023, neutral): “Similarly, InstructTTS [2] uses discrete diffusion models to generated discrete tokens, then uses audio codec model to recover the waveform.”
• Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, background): “2) non-autoregressive (NAR) [42, 35]: which shows advantages in training stability but has been concerned with the requirement of an alignment model and “averaged” duration being predicted.”
• HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “Meanwhile, there are text prompt-based style generation models which can describe a speaking or voice style from text descriptions [61], [62], [63].”
• TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation (2022, background): “…works bring NAR model into other sequence-to-sequence tasks (Cui et al., 2021; Ye et al., 2023; Huang et al., 2022c; Yang et al., 2022b), such as video caption (Yang et al., 2019), speech recognition (Chen et al., 2020) and speech synthesis (Ye et al., 2022; Huang et al., 2022d; Yang et al., 2023).”