A Survey on Neural Speech Synthesis

By Xu Tan, Tao Qin, Frank K. Soong, Tie-Yan Liu. 2021

478 citations recorded by the source corpus.

Text to speech (TTS), or speech synthesis, which aims to synthesize intelligible and natural speech given text, is a hot research topic in speech, language, and machine learning communities and has broad applications in the industry. As the development of deep learning and artificial intelligence, neural network-based TTS has significantly improved the quality of synthesized speech in recent years. In this paper, we conduct a comprehensive survey on neural TTS, aiming to provide a good understanding of current research and future trends. We focus on the key components in neural TTS, including text analysis, acoustic models and vocoders, and several advanced topics, including fast TTS, low-resource TTS, robust TTS, expressive TTS, and adaptive TTS, etc. We further summarize resources related to TTS (e.g., datasets, opensource implementations) and discuss future research directions. This survey can serve both academic researchers and industry practitioners working on TTS.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “To achieve such higher levels of expressiveness, TTS systems historically had to rely on labeled datasets for specific speech phenomena; and even so, achieving human-like prosody for certain types of textual inputs has remained elusive [30].”
UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, background): “Prior works on audio generation tasks are commonly task-specific: their designs heavily leverage domain knowledge and their usage is restricted to fixed setups (Tan et al., 2021; Luo & Mesgarani, 2019; Zmolikova et al., 2023; Huang et al., 2021b; Cho et al., 2021).”
StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, background): “Text-to-speech (TTS) synthesis has seen significant advancements in recent years, with numerous applications such as virtual assistants, audiobooks, and voice-over narration benefiting from increasingly natural and expressive synthetic speech [1, 2].”
Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, background): “For decades, the increasing demand for personalized speech generation in various applications has posed challenges for TTS models [53], especially in zero-shot multi-speaker scenarios regarding domain shifts.”
NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, background): “Text-to-speech systems [2, 3, 4, 5, 6, 8, 9, 10, 11] aim to generate natural voice with both high quality and diversity.”
DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders (2022, methodology): “Popular TTS models [1] are based on a typical two-stage system consisting of an acoustic model and a vocoder.”
AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios (2022, background): “Neural text to speech (TTS) [1] models can synthesize high quality human voice when being trained with a large amount of single-speaker or multi-speaker datasets [2, 3, 4, 5, 6, 7, 8, 9].”
Diffsound: Discrete Diffusion Model for Text-to-Sound Generation (2022, background): “Most of the text-to-speech (TTS) methods [34], [35], [41] directly learn the mapping from text to wave samples or raw spectrogram pixels for the reason that the synthesized speech’s content relies on the words of text.”
InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, methodology): “Most text-to-speech (TTS) methods [2], [40], [53] directly learn the mapping from text to mel-spectrogram in continuous space.”
DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021 (2021, methodology): “Typical TTS systems consist of three key components [1]: text analysis, acoustic model, and vocoder.”
Simple and Controllable Music Generation (2023, background): “Recent advancements in self-supervised audio representation learning [Balestriero et al., 2023], sequential modeling [Touvron et al., 2023], and audio synthesis [Tan et al., 2021] provide the conditions to develop such models.”
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021, background): “Along with automatic speech recognition (ASR) and machine translation (MT), these language technologies have advanced rapidly over the past few years (Tan et al., 2021).”
DiffVoice: Text-to-Speech with Latent Diffusion (2023, methodology): “Similar to Deep Voice [23] and TalkNet [24], we rely on a CTC based ASR model trained on phoneme sequences to obtain the alignment [3] between w and y .”
CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model (2023, background): “Speech synthesis [Tan et al., 2021] aims to produce realistic audio of humans and has broadly included text-to-speech (TTS) [Taylor, 2009] and singing voice synthesis (SVS) [Nishimura et al.”
StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis (2022, unknown): “However, synthesizing expressive speech that captures the full range of prosodic, temporal, and spectral features, also known as paralinguistic information, remains a challenge [4].”
XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model (2024, background): “Speakers who have a voice that differs greatly from those seen in training also become a challenge [35].”
AdaSpeech 3: Adaptive Text to Speech for Spontaneous Style (2021, background): “Text to speech (TTS) aims to synthesize intelligible and nat-ural speech [1, 2, 3, 4, 5, 6].”