NaturalSpeech: End-to-End Text-to-Speech Synthesis With Human-Level Quality

By Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Yanqing Liu, Yichong Leng, Yuanhao Yi, Lei He, Frank K. Soong, Tao Qin and 2 more. 2022

351 citations recorded by the source corpus.

Text-to-speech (TTS) has made rapid progress in both academia and industry in recent years. Some questions naturally arise that whether a TTS system can achieve human-level quality, how to define/judge that quality, and how to achieve it. In this paper, we answer these questions by first defining the human-level quality based on the statistical significance of subjective measure and introducing appropriate guidelines to judge it, and then developing a TTS system called NaturalSpeech that achieves human-level quality on benchmark datasets. Specifically, we leverage a variational auto-encoder (VAE) for end-to-end text-to-waveform generation, with several key modules to enhance the capacity of the prior from text and reduce the complexity of the posterior from speech, including phoneme pre-training, differentiable duration modeling, bidirectional prior/posterior modeling, and a memory mechanism in VAE. Experimental evaluations on the popular LJSpeech dataset show that our proposed NaturalSpeech achieves <inline-formula><tex-math notation="LaTeX">$-0.01$</tex-math><alternatives><mml:math><mml:mrow><mml:mo>-</mml:mo><mml:mn>0</mml:mn><mml:mo>.</mml:mo><mml:mn>01</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="tan-ieq1-3356232.gif"/></alternatives></inline-formula> CMOS (comparative mean opinion score) to human recordings at the sentence level, with Wilcoxon signed rank test at p-level <inline-formula><tex-math notation="LaTeX">$p \gg 0.05$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>p</mml:mi><mml:mo>≫</mml:mo><mml:mn>0</mml:mn><mml:mo>.</mml:mo><mml:mn>05</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="tan-ieq2-3356232.gif"/></alternatives></inline-formula>, which demonstrates no statistically significant difference from human recordings for the first time.

Subfields: Artificial Intelligence, Natural Language Generation, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Xu Tan
  2. Tao Qin
  3. Sheng Zhao
  4. Tie-Yan Liu
  5. Lei He
  6. Nima Mesgarani
  7. Yanqing Liu
  8. Cong Han

In Natural Language Generation

  1. Yanqing Liu
  2. Sheng Zhao
  3. Lei He

In Speech & Audio

  1. Xu Tan
  2. Lei He
  3. Sheng Zhao
  4. Tao Qin
  5. Yanqing Liu
  6. Nima Mesgarani
  7. Cong Han
  8. Jinyu Li

In Signal Processing

  1. Xu Tan
  2. Tao Qin
  3. Lei He
  4. Sheng Zhao
  5. Nima Mesgarani
  6. Cong Han
  7. Yanqing Liu
  8. Jinyu Li

Citations in the corpus