Deep Voice 2: Multi-Speaker Neural Text-to-Speech

By Andrew Gibiansky, Sercan Ö. Arik, G. Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, Yanqi Zhou. 2017

528 citations recorded by the source corpus.

We introduce a technique for augmenting neural text-to-speech (TTS) with lowdimensional trainable speaker embeddings to generate different voices from a single model. As a starting point, we show improvements over the two state-ofthe-art approaches for single-speaker neural TTS: Deep Voice 1 and Tacotron. We introduce Deep Voice 2, which is based on a similar pipeline with Deep Voice 1, but constructed with higher performance building blocks and demonstrates a significant audio quality improvement over Deep Voice 1. We improve Tacotron by introducing a post-processing neural vocoder, and demonstrate a significant audio quality improvement. We then demonstrate our technique for multi-speaker speech synthesis for both Deep Voice 2 and Tacotron on two multi-speaker TTS datasets. We show that a single neural TTS system can learn hundreds of unique voices from less than half an hour of data per speaker, while achieving high audio quality synthesis and preserving the speaker identities almost perfectly.

Subfields: Artificial Intelligence, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Yu Zhang
6. Hung-yi Lee
7. Junichi Yamagishi
8. Lei He

In Speech & Audio

1. Xu Tan
2. Tao Qin
3. Lei He
4. Hung-yi Lee
5. Tie-Yan Liu
6. Yu Zhang
7. Sheng Zhao
8. Junichi Yamagishi

Citations in the corpus

• Jukebox: A Generative Model for Music (2020, background): “To generate multiple voices, text-to-speech models can also condition on the speaker identity (Oord et al., 2016; Gibiansky et al., 2017; Jia et al., 2018) as well as text prompt.”
• Waveglow: A Flow-based Generative Network for Speech Synthesis (2018, background): “Since then, several auto-regressive models have attempted to speed up inference while retaining quality [6, 20, 21].”
• Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions (2017, methodology): “In this paper, we describe a unified, entirely neural approach to speech synthesis that combines the best of the previous approaches: a sequence-to-sequence Tacotron-style model [12] that generates mel spectrograms, followed by a modified WaveNet vocoder [10, 15].”
• Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis (2018, methodology): “Results are shown in Table 1, comparing the proposed model to baseline multispeaker models that utilize a lookup table of speaker embeddings similar to [8, 13], but otherwise have identical architectures to the proposed synthesizer network.”
• Semi-Supervised Neural Architecture Search (2020, methodology): “Moreover, unlike many other sequence learning tasks (e.g., neural machine translation, language modeling) where the Trans-former model (Vaswani et al., 2017) is the dominate architecture, RNN based Tacotron (Wang et al., 2017; Shen et al., 2018), CNN based Deep Voice (Arik et al., 2017; Gibian-sky e”
• AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios (2022, background): “Neural text to speech (TTS) [1] models can synthesize high quality human voice when being trained with a large amount of single-speaker or multi-speaker datasets [2, 3, 4, 5, 6, 7, 8, 9].”
• Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech (2021, background): “trained with a large amount of high-quality single-speaker speech data [1], [2], [3] or synthesize a set of voices with multi-speaker corpora [4], [5], [6], [7].”
• A Survey on Neural Speech Synthesis (2021, methodology): “The original WaveNet, as well as some following works that leverage WaveNet as vocoder [8, 84], generate speech waveform conditioned on linguistic features, while WaveNet can be easily adapted to condition on linear-spectrograms [84] and mel-spectrograms [330, 264, 297].”
• AdaSpeech: Adaptive Text to Speech for Custom Voice (2021, background): “Text to speech (TTS) aims to synthesize natural and intelligible voice from text, and attracts a lot of interests in machine learning community (Arik et al., 2017; Wang et al., 2017; Gibiansky et al., 2017; Ping et al., 2018; Shen et al., 2018; Ren et al., 2019).”
• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech (2019, methodology): “The VCTK corpus [20] is popular for experimenting with multi-speaker TTS [6, 8, 11, 14, 29–31] as it contains studio quality speech data from multiple speakers.”
• Neural Voice Cloning with a Few Samples (2018, background): “Speaker embeddings have been shown to effectively capture speaker characteristics with low-dimensional vectors [Arik et al., 2017b, Ping et al., 2018].”
• Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron (2018, methodology): “We follow a scheme similar to (Arık et al., 2017) to model multiple speakers.”
• Fitting New Speakers Based on a Short Untranscribed Sample (2018, methodology): “The recent neural TTS systems include the Deep Voice systems DV1 (Arik et al., 2017b), DV2 (Arik et al., 2017a) and DV3 (Ping et al., 2018), WaveNet (Oord et al., 2016) and Parallel WaveNet (van den Oord et al., 2017), Char2Wav (Sotelo et al., 2017), Tacotron (Wang et al., 2017) and Tacotron2 (Shen ”
• Deep Voice 3: 2000-Speaker Neural Text-to-Speech (2017, methodology): “There is not yet a consensus on the optimal neural network architecture for TTS, however, sequence-to-sequence models (Wang et al., 2017; Sotelo et al., 2017; Arık et al., 2017) have shown to be quite promising.”
• ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech (2018, methodology): “We use the similar WaveNet architecture detailed in Arık et al. (2017a).”
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis (2019, background): “In the case of speech, aligned linguistic features (Van Den Oord et al., 2016) and mel-spectograms (Shen et al., 2018; Gibiansky et al., 2017) are two commonly used intermediate representations.”