NaturalSpeech: End-to-End Text-to-Speech Synthesis With Human-Level Quality
By Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Yanqing Liu, Yichong Leng, Yuanhao Yi, Lei He, Frank K. Soong, Tao Qin and 2 more. 2022
351 citations recorded by the source corpus.
Text-to-speech (TTS) has made rapid progress in both academia and industry in recent years. Some questions naturally arise that whether a TTS system can achieve human-level quality, how to define/judge that quality, and how to achieve it. In this paper, we answer these questions by first defining the human-level quality based on the statistical significance of subjective measure and introducing appropriate guidelines to judge it, and then developing a TTS system called NaturalSpeech that achieves human-level quality on benchmark datasets. Specifically, we leverage a variational auto-encoder (VAE) for end-to-end text-to-waveform generation, with several key modules to enhance the capacity of the prior from text and reduce the complexity of the posterior from speech, including phoneme pre-training, differentiable duration modeling, bidirectional prior/posterior modeling, and a memory mechanism in VAE. Experimental evaluations on the popular LJSpeech dataset show that our proposed NaturalSpeech achieves <inline-formula><tex-math notation="LaTeX">$-0.01$</tex-math><alternatives><mml:math><mml:mrow><mml:mo>-</mml:mo><mml:mn>0</mml:mn><mml:mo>.</mml:mo><mml:mn>01</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="tan-ieq1-3356232.gif"/></alternatives></inline-formula> CMOS (comparative mean opinion score) to human recordings at the sentence level, with Wilcoxon signed rank test at p-level <inline-formula><tex-math notation="LaTeX">$p \gg 0.05$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>p</mml:mi><mml:mo>≫</mml:mo><mml:mn>0</mml:mn><mml:mo>.</mml:mo><mml:mn>05</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="tan-ieq2-3356232.gif"/></alternatives></inline-formula>, which demonstrates no statistically significant difference from human recordings for the first time.
Subfields: Artificial Intelligence, Natural Language Generation, Speech & Audio, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Natural Language Generation
In Speech & Audio
In Signal Processing
Citations in the corpus
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “Traditional TTS systems [1, 2, 3, 4] are typically trained on limited datasets recorded in studios, and thus fail to support high-quality zero-shot speech synthesis.”
- StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “Some previous works have made significant progress towards human-level performance [3, 4, 5].”
- High-Fidelity Audio Compression with Improved RVQGAN (2023, methodology): “This formulation, with continuous latent variables and training an expressive prior using normalizing flows has been quite successful for speech synthesis [17, 35].”
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “Compared to its previous version NaturalSpeech [11], NaturalSpeech 2 has the following connections and distinctions.”
- Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, neutral): “Recent years have witnessed significant advancements in end-to-end text-to-speech (TTS) synthesis, and the quality of synthesized speech is even close to human parity [Li et al., 2019, Ren et al., 2019, Tan et al., 2022].”
- DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders (2022, neutral): “To solve the issues in two-stage cascaded TTS systems, fully end-to-end TTS models [6, 3, 7, 8, 9] are developed recently.”
- StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis (2022, neutral): “T EXT-TO-SPEECH (TTS), also known as speech synthesis, has made significant strides with the advent of deep learning, producing increasingly human-like synthetic speech [1], [2], [3].”
- PromptTTS 2: Describing and Generating Voices with Text Prompt (2023, background): “Some TTS systems have achieved a level of quality comparable to that of single-speaker recordings (Tan et al., 2022), and large-scale TTS systems have been developed for multi-speaker scenarios (Wang et al.”
- HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, positive): “NaturalSpeech [24] achieved human-level quality in a single speaker TTS by introducing a bidirectional normalizing flow and phoneme pretraining.”
- AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining (2023, methodology): “2 Phoneme Encoder is a widely adopted module in text-to-speech research for extracting helpful information regarding phonemes [2], [11], which are the smallest units of sound in a language that can distinguish one word from another [75].”
- AudioLDM: Text-to-Audio Generation with Latent Diffusion Models (2023, background): “ative models (Oord et al., 2016; Ho et al., 2020; Song et al., 2021; Tan et al., 2022), either unconditional or conditioned on other modalities (Kreuk et al.”
- F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (2024, background): “Recent research in Text-to-Speech (TTS) has experienced great advancement [1, 2, 3, 4, 5, 6, 7, 8].”
- Prompttts: Controllable Text-To-Speech With Text Descriptions (2022, background): “Beyond text and image generation, there is little research on prompt-based guidance for text-to-speech (TTS) synthesis [7, 8] with different styles such as pitch, speaking speed, and emotion.”
- PHONEME-LEVEL BERT FOR ENHANCED PROSODY OF TEXT-TO-SPEECH WITH GRAPHEME PREDICTIONS (2023, background): “Text-to-speech (TTS) has seen significant progress in recent years, and the most recent works are shown to synthesize speech indistinguishable from natural human speech for in-distribution texts evaluated subjectively by human raters [1].”
- FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model (2023, methodology): “The acoustic model in traditional neural TTS is designed to predict continuous speech features, for example, mel-spectrograms used by TransformerTTS, FastSpeech, and DelightfulTTS [12, 13, 16] or self-learned acoustic features used by NaturalSpeech and DelightfulTTS 2 [22, 18] from linear spectrum o”