Deep Voice: Real-time Neural Text-to-Speech

By Sercan Ö. Arik, Mike Chrzanowski, Adam Coates, G. Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Ng, Jonathan Raiman and 2 more. International Conference on Machine Learning, 2017

666 citations recorded by the source corpus.

We present Deep Voice, a production-quality text-to-speech system constructed entirely from deep neural networks. Deep Voice lays the groundwork for truly end-to-end neural speech synthesis. The system comprises five major building blocks: a segmentation model for locating phoneme boundaries, a grapheme-to-phoneme conversion model, a phoneme duration prediction model, a fundamental frequency prediction model, and an audio synthesis model. For the segmentation model, we propose a novel way of performing phoneme boundary detection with deep neural networks using connectionist temporal classification (CTC) loss. For the audio synthesis model, we implement a variant of WaveNet that requires fewer parameters and trains faster than the original. By using a neural network for each component, our system is simpler and more flexible than traditional text-to-speech systems, where each component requires laborious feature engineering and extensive domain expertise. Finally, we show that inference with our system can be performed faster than real time and describe optimized WaveNet inference kernels on both CPU and GPU that achieve up to 400x speedups over existing implementations.

Subfields: Artificial Intelligence, Natural Language Processing

View Paper: https://arxiv.org/abs/1702.07825

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Xu Tan
3. Tie-Yan Liu
4. Frank K. Soong
5. Yu Zhang
6. Junichi Yamagishi
7. Yoshua Bengio
8. Quoc V. Le

In Natural Language Processing

1. Tao Qin
2. Tie-Yan Liu
3. Quoc V. Le
4. Kyunghyun Cho
5. Yoshua Bengio
6. Graham Neubig
7. Xu Tan
8. Mohit Bansal

Citations in the corpus

• Jukebox: A Generative Model for Music (2020, methodology): “, 2009), and often complex pipelines as described in (Arık et al., 2017).”
• Waveglow: A Flow-based Generative Network for Speech Synthesis (2018, background): “The first step transforms the text into time-aligned features, such as a mel-spectrogram [4, 5], or F0 frequencies and other linguistic features [2, 6].”
• Efficient Neural Audio Synthesis (2018, background): “We found that on low-power platforms such as mobile CPUs even fast vectorized approximations (Arik et al., 2017) to tanh and sigmoid are too slow.”
• Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions (2017, background): “WaveNet [8], a generative model of time domain waveforms, produces audio quality that begins to rival that of real human speech and is already used in some complete TTS systems [9, 10, 11].”
• Tacotron: Towards End-to-End Speech Synthesis (2017, methodology): “It could be used to predict alternative targets such as vocoder parameters, or as a WaveNetlike neural vocoder [9, 13, 10] that synthesizes waveform samples directly.”
• Semi-Supervised Neural Architecture Search (2020, methodology): “Moreover, unlike many other sequence learning tasks (e.g., neural machine translation, language modeling) where the Trans-former model (Vaswani et al., 2017) is the dominate architecture, RNN based Tacotron (Wang et al., 2017; Shen et al., 2018), CNN based Deep Voice (Arik et al., 2017; Gibian-sky e”
• ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, methodology): “Later, end-to-end neural TTS models were introduced, which synthe-size Mel spectrograms and employ a vocoder (Oord et al., 2017; Prenger et al., 2019) for speech synthesis (Wang et al., 2017; Arık et al., 2017; Ren et al., 2019).”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “Text-to-speech models usually generate mel-spectrogram from text [59, 2, 35, 48, 29, 47, 36, 22] and then synthesize speech waveform from the generated mel-spectrogram using a separately pre-trained vocoder [41, 31, 62, 20], or directly generate waveform from text in an end-to-end manner [45, 15,…”
• A Survey on Neural Speech Synthesis (2021, methodology): “Hybrid DeepVoice 1/2 [8, 84], Tacotron [376], DurIAN [411], LightSpeech [214], Wave-Tacotron [379]”
• AdaSpeech: Adaptive Text to Speech for Custom Voice (2021, background): “Text to speech (TTS) aims to synthesize natural and intelligible voice from text, and attracts a lot of interests in machine learning community (Arik et al., 2017; Wang et al., 2017; Gibiansky et al., 2017; Ping et al., 2018; Shen et al., 2018; Ren et al., 2019).”
• TTS-Portuguese Corpus: a corpus for speech synthesis in Brazilian Portuguese (2020, neutral): “Models based on Recurrent Neural Networks such as Tacotron [16], Tacotron 2 [17], Deep Voice 1 [26] and Deep Voice 2 [27] have gained prominence, but as these models use recurrent layers they have high computational costs.”
• Sample Efficient Adaptive Text-to-Speech (2018, background): “, 2017) (building on (Arık et al., 2017)) introduced a multi-speaker variation of Tacotron that learns a low-dimensional embedding for each speaker, which was further extended in DeepVoice 3 (Ping et al.”
• Neural Voice Cloning with a Few Samples (2018, background): “Speaker embeddings have been shown to effectively capture speaker characteristics with low-dimensional vectors [Arik et al., 2017b, Ping et al., 2018].”
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis (2018, background): “The past few years have seen exciting developments in the use of deep neural networks to synthesize natural-sounding human speech (Zen et al., 2016; van den Oord et al., 2016; Wang et al., 2017a; Arik et al., 2017; Taigman et al., 2017; Shen et al., 2017).”
• Fitting New Speakers Based on a Short Untranscribed Sample (2018, methodology): “The recent neural TTS systems include the Deep Voice systems DV1 (Arik et al., 2017b), DV2 (Arik et al., 2017a) and DV3 (Ping et al., 2018), WaveNet (Oord et al., 2016) and Parallel WaveNet (van den Oord et al., 2017), Char2Wav (Sotelo et al., 2017), Tacotron (Wang et al., 2017) and Tacotron2 (Shen ”
• Deep Voice 3: 2000-Speaker Neural Text-to-Speech (2017, background): “ents, and higher quality synthesized speech. There is not yet a consensus on the optimal neural network architecture for TTS. However, sequence-to-sequence models (Wang et al.,2017;Sotelo et al.,2017;Arık et al., 2017) have shown to be quite promising. In this paper, we propose a novel fully-convolu”