Deep Voice 3: 2000-Speaker Neural Text-to-Speech

By Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan Ö. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, John Miller. 2017

322 citations recorded by the source corpus.

We present Deep Voice 3, a fully-convolutional attention-based neural text-to-speech (TTS) system. Deep Voice 3 matches state-of-the-art neural speech synthesis systems in naturalness while training ten times faster. We scale Deep Voice 3 to data set sizes unprecedented for TTS, training on more than eight hundred hours of audio from over two thousand speakers. In addition, we identify common error modes of attention-based speech synthesis networks, demonstrate how to mitigate them, and compare several different waveform synthesis methods. We also describe how to scale inference to ten million queries per day on one single-GPU server.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Junichi Yamagishi
6. Yu Zhang
7. Hung-yi Lee
8. Dong Yu

In Natural Language Processing

1. Tie-Yan Liu
2. Tao Qin
3. Junichi Yamagishi
4. Xu Tan
5. Yoshua Bengio
6. Lei He
7. Yu Zhang
8. Quoc V. Le

In Speech & Audio

1. Xu Tan
2. Tao Qin
3. Junichi Yamagishi
4. Tie-Yan Liu
5. Lei He
6. Sheng Zhao
7. Yi Ren
8. Yu Zhang

Citations in the corpus

• Jukebox: A Generative Model for Music (2020, methodology): “More recent works like Deep Voice 3 (Ping et al., 2018), Tacotron 2 (Shen et al., 2018), and Char2Wav (Sotelo et al., 2017) learn speech synthesis end-to-end using sequence-to-sequence architecture (Sutskever et al., 2014).”
• Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions (2017, methodology): “We ran a separate rating experiment on the custom 100-sentence test set from Appendix E of [11], obtaining a MOS of 4.”
• Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis (2018, methodology): “Deep Voice 3 [13] proposed a fully convolutional encoder-decoder architecture which scaled up to support over 2,400 speakers from LibriSpeech [12].”
• Semi-Supervised Neural Architecture Search (2020, background): “…between the encoder and decoder is an important influence factor on the quality of synthesized speech in previous works (Ren et al., 2019a; Wang et al., 2017; Shen et al., 2018; Li et al., 2019; Ping et al., 2017), and misalignment can be observed for most mistakes (e.g., skipping and repeating).”
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “Speech Representations in TTS. Traditional works propose using prior-based speech representation such as raw waveform [26, 27, 28] or mel-spectrogram [29, 30, 3, 31].”
• AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios (2022, background): “Neural text to speech (TTS) [1] models can synthesize high quality human voice when being trained with a large amount of single-speaker or multi-speaker datasets [2, 3, 4, 5, 6, 7, 8, 9].”
• YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone (2021, positive): “ZS-TTS using deep learning was first proposed by [5] which extended the DeepVoice 3 method [6].”
• Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech (2021, methodology): “[8] uses DeepVoice3 [5] as the base model architecture.”
• A Survey on Neural Speech Synthesis (2021, methodology): “, Tacotron 1/2 [376, 297], Deep Voice 3 [264], and FastSpeech 1/2 [284, 286]) are proposed to simplify text analysis modules and directly take character/phoneme sequences as input, and simplify acoustic features with mel-spectrograms.”
• Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, background): “Currently, neural TTS models with a two-stage pipeline can synthesize human-like speech (Oord et al., 2016; Ping et al., 2018; Shen et al., 2018).”
• AdaSpeech: Adaptive Text to Speech for Custom Voice (2021, background): “TTS models can synthesize natural human voice when training with a large amount of high-quality and single-speaker recordings (Ito, 2017), and has been extended to multi-speaker scenarios (Gibiansky et al., 2017; Ping et al., 2018; Zen et al., 2019; Chen et al., 2020) using multi-speaker corpora (Pa”
• TTS-Portuguese Corpus: a corpus for speech synthesis in Brazilian Portuguese (2020, methodology): “[18] proposed a fully convolutional model for speech synthesis and compared three different vocoders: Griffin-Lim [28], WORLD Vocoder [29] and WaveNet [30].”
• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech (2019, background): “Although this corpus was originally designed for automaticspeechrecognition (ASR)research, ithasbeenusedin various text-to-speech (TTS) research projects [7,8,11] thanks to its attractive properties, such as a non-restrictive license, a large amount of data, and wide speaker diversity.”
• Sample Efficient Adaptive Text-to-Speech (2018, methodology): “The most similar approached to our work was proposed by Arik et al. (2018) for the DeepVoice 3 model.”
• Neural Voice Cloning with a Few Samples (2018, methodology): “MULTI-SPEAKER GENERATIVE MODEL Our multi-speaker generative model is based on the convolutional sequence-to-sequence architecture proposed in (Ping et al., 2017), with similar hyperparameters and GriffinLim vocoder.”
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis (2018, methodology): “For a baseline, we train a multi-speaker Tacotron similar to (Ping et al., 2017), using video IDs as a proxy for speaker labels.”
• Fitting New Speakers Based on a Short Untranscribed Sample (2018, methodology): “The recent neural TTS systems include the Deep Voice systems DV1 (Arik et al., 2017b), DV2 (Arik et al., 2017a) and DV3 (Ping et al., 2018), WaveNet (Oord et al., 2016) and Parallel WaveNet (van den Oord et al., 2017), Char2Wav (Sotelo et al., 2017), Tacotron (Wang et al., 2017) and Tacotron2 (Shen ”
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech (2020, methodology): “Previous neural TTS models such as Tacotron [25], Tacotron 2 [21], Deep Voice 3 [17] and Transformer TTS [10] first generate mel-spectrograms autoregressively from text and then synthesize speech from the generated mel-spectrograms using a separately trained vocoder (e.”
• Adversarial Audio Synthesis (2018, methodology): “, 2016) and Griffin-Lim (Griffin & Lim, 1984), or trained neural vocoders (Sotelo et al., 2017; Shen et al., 2018; Ping et al., 2018).”
• High Fidelity Speech Synthesis with Adversarial Networks (2019, neutral): “Models like Deep Voice 2 & 3 (Gibiansky et al., 2017; Ping et al., 2018) and Tacotron 2 (Shen et al.”