NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers

By Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, Jiang Bian. 2023

394 citations recorded by the source corpus.

Scaling text-to-speech (TTS) to large-scale, multi-speaker, and in-the-wild datasets is important to capture the diversity in human speech such as speaker identities, prosodies, and styles (e.g., singing). Current large TTS systems usually quantize speech into discrete tokens and use language models to generate these tokens one by one, which suffer from unstable prosody, word skipping/repeating issue, and poor voice quality. In this paper, we develop NaturalSpeech 2, a TTS system that leverages a neural audio codec with residual vector quantizers to get the quantized latent vectors and uses a diffusion model to generate these latent vectors conditioned on text input. To enhance the zero-shot capability that is important to achieve diverse speech synthesis, we design a speech prompting mechanism to facilitate in-context learning in the diffusion model and the duration/pitch predictor. We scale NaturalSpeech 2 to large-scale datasets with 44K hours of speech and singing data and evaluate its voice quality on unseen speakers. NaturalSpeech 2 outperforms previous TTS systems by a large margin in terms of prosody/timbre similarity, robustness, and voice quality in a zero-shot setting, and performs novel zero-shot singing synthesis with only a speech prompt. Audio samples are available at https://speechresearch.github.io/naturalspeech2.

Subfields: Artificial Intelligence, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Xu Tan
3. Dong Yu
4. Tao Qin
5. Tie-Yan Liu
6. Zhou Zhao
7. Nima Mesgarani
8. Sheng Zhao

In Generative Models

1. Zhou Zhao
2. Yi Ren
3. Rongjie Huang
4. Jinglin Liu
5. Yossi Adi
6. Wei-Ning Hsu
7. Zhijun Liu
8. Matt Le

In Speech & Audio

1. Jinyu Li
2. Xu Tan
3. Lei He
4. Yi Ren
5. Zhou Zhao
6. Rongjie Huang
7. Sheng Zhao
8. Jinglin Liu

In Signal Processing

1. Jinyu Li
2. Dong Yu
3. Xu Tan
4. Nima Mesgarani
5. Zhuo Chen
6. Rongjie Huang
7. Zhou Zhao
8. Shinji Watanabe

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “StyleTTS 2 [53], UniCATS [54], NaturalSpeech 2 [55] and NaturalSpeech 3 [56] employed diffusion model [57] for the prompt-conditioned text to speech synthesis.”
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “StyleTTS 2 [Li et al., 2024], UniCATS [Du et al., 2024a], NaturalSpeech 2 [Shen et al., 2023] and NaturalSpeech 3 [Ju et al., 2024] used diffusion model [Ho et al., 2020] for the prompt-conditioned text to speech synthesis.”
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “2 ) Natural-Speech 2 [5].”
• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, methodology): “Automated objective evaluations to test TTS robustness, especially issues with missed, duplicated or hallucinated synthesis [20], and speaker similarity.”
• ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, methodology): “By leveraging diffusion models and their variants (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song and Ermon, 2020; Song et al., 2021; Lipman et al., 2023), many works have successfully applied them to the audio domain (Popov et al., 2021; Huang et al., 2022, 2023; Shen et al., 2023).”
• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, methodology): “…auditory perception (Shadle & Damper, 2001) and machine learning (Wang et al., 2016) research etc; (4) they use different generative model, such as diffusion model (Huang et al., 2023a; Shen et al., 2023; Wang et al., 2023b), flow (Le et al., 2023), Seq2Seq (Ren et al., 2020; Liu et al., 2021).”
• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, methodology): “These tasks include voice or style conversion (Make-A-Voice [18], NaturalSpeech2 [31],”
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, background): “4) Voicebox is compatible with any continuous features including Encodec embeddings.”
• UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “NaturalSpeech 2 (Shen et al. 2023) employs a typical diffusion model to generate discrete acoustic tokens as continuous features.”
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “Recent advancements have proven the effectiveness of large-scale self-supervised speech language models (SLMs) in enhancing text-tospeech (TTS) quality [34, 35, 36, 37] and speaker adaptation [8, 38, 29, 39].”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “Monotonic alignment with neural codec models [14] as the intermediate representation and model it with autoregressive language models (LM) [58] or diffusion models [50].”
• Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, background): “Current large-scale voice synthesis [18, 39, 35] systems leverage the codec models to predict the intermediate representation instead of waveforms, which can be clustered into two groups: 1) autoregressive (AR) [4, 18, 39, 46]: which integrates implicit duration modeling but suffers from error…”
• RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis (2024, methodology): “The result of NaturalSpeech 2 is from Shen et al. [25].”
• PromptTTS 2: Describing and Generating Voices with Text Prompt (2023, methodology): “which takes a reference speech as input and outputs a reference hidden (Shen et al., 2023; Wang et al., 2023).”
• CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model (2023, methodology): “For audio quality, our teacher model achieved the highest MOS and Grad-TTS ranked second because our teacher model is based on the design of Grad-TTS, but we adopt better choices on drift and diffusion coefficients in SDE.”
• F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (2024, background): “The synthesized speech can achieve high fidelity and naturalness that they are almost indistinguishable from human speech [11, 12, 13, 14].”
• Mega-TTS 2: Zero-Shot Text-to-Speech with Arbitrary Length Speech Prompts (2023, methodology): “NeturalSpeech 2 (Shen et al., 2023) uses continuous vectors instead of discrete neural codec tokens and introduces in-context learning to latent diffusion model (Rombach et al.”
• HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, neutral): “The details are described in the following subsections.”
• StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis (2022, neutral): “…based on variational inference [1], [5], [6], [7], flow-based modeling [1], [8], [9], explicit modeling of pitch, duration and energy [10], [11], [12], autoregressive language modeling (LM) objectives [13], [14], [15], and using external prosody encoder [16], [17], [18], the production of…”