Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis

By Ye Jia, Yu Zhang, Ron J. Weiss, Quan Wang, Jonathan Shen, Fei Ren, Z. Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez-Moreno and 1 more. Neural Information Processing Systems, 2018

963 citations recorded by the source corpus.

We describe a neural network-based system for text-to-speech (TTS) synthesis that is able to generate speech audio in the voice of many different speakers, including those unseen during training. Our system consists of three independently trained components: (1) a speaker encoder network, trained on a speaker verification task using an independent dataset of noisy speech from thousands of speakers without transcripts, to generate a fixed-dimensional embedding vector from seconds of reference speech from a target speaker; (2) a sequence-to-sequence synthesis network based on Tacotron 2, which generates a mel spectrogram from text, conditioned on the speaker embedding; (3) an auto-regressive WaveNet-based vocoder that converts the mel spectrogram into a sequence of time domain waveform samples. We demonstrate that the proposed model is able to transfer the knowledge of speaker variability learned by the discriminatively-trained speaker encoder to the new task, and is able to synthesize natural speech from speakers that were not seen during training. We quantify the importance of training the speaker encoder on a large and diverse speaker set in order to obtain the best generalization performance. Finally, we show that randomly sampled speaker embeddings can be used to synthesize speech in the voice of novel speakers dissimilar from those used in training, indicating that the model has learned a high quality speaker representation.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

View Paper: https://arxiv.org/abs/1806.04558

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Xu Tan
3. Tie-Yan Liu
4. Jinyu Li
5. Yu Zhang
6. Dong Yu
7. Shinji Watanabe
8. Hung-yi Lee

In Speech & Audio

1. Jinyu Li
2. Xu Tan
3. Yu Zhang
4. Shinji Watanabe
5. Hung-yi Lee
6. Tao Qin
7. Junichi Yamagishi
8. Wei-Ning Hsu

In Signal Processing

1. Jinyu Li
2. Xu Tan
3. Junichi Yamagishi
4. Dong Yu
5. Tao Qin
6. Tie-Yan Liu
7. Shinji Watanabe
8. Yu Zhang

Citations in the corpus

• Jukebox: A Generative Model for Music (2020, background): “To generate multiple voices, text-to-speech models can also condition on the speaker identity (Oord et al., 2016; Gibiansky et al., 2017; Jia et al., 2018) as well as text prompt.”
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling (2019, background): “To date, this framework has produced a number of state-of-the-art results in machine translation [2, 21, 23], speech recognition [3, 4, 5, 10, 11, 12, 13, 14, 16, 15, 20, 22], speech synthesis [6, 8, 19], and speech translation [7, 9].”
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, methodology): “A speaker encoding based system contains a speaker encoder and a TTS component, where the speaker encoder could be pre-trained on the speaker verification task [Jia et al., 2018].”
• FedSpeech: Federated Text-to-Speech with Continual Learning (2021, background): “…estimating the speaker features from the latent space and protect privacy, we introduce a private speaker module which is a trainable lookup table [ Jia et al. , 2018 ] taking speaker’s identity number S id as input and generating the speaker representation R = { r 1 , r 2 , ..., r n } , where n…”
• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, background): “Early studies in zero-shot TTS employed fixeddimensional speaker embeddings [11], [12], [13], [14].”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, background): “Previous approaches can be categorized into speaker adaptation [13, 11, 60, 23] and speaker encoding [25, 1, 26, 61] methods.”
• PolyVoice: Language Models for Speech to Speech Translation (2023, background): “Current approaches, such as speaker embeddings (Jia et al., 2019) and multi-speaker TTS systems (Jia et al., 2018), have made some progress in this direction, but they are still far from for practical requirements.”
• Grad-StyleSpeech: Any-Speaker Adaptive Text-to-Speech Synthesis with Diffusion Models (2022, background): “On the other hand, recent works [12, 14, 15] have focused on a zero-shot approach in which transcribed samples and the additional fine-tuning stage are not necessarily required to adapt to the unseen speaker, thanks to their neural encoder that encode any speech into the latent vector.”
• A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing (2022, neutral): “FastSpeech 2 3.34 ± 0.11 3.85 ± 0.11 +GST (Wang et al., 2018) 3.27 ± 0.11 3.72 ± 0.11 A 3 T 3.63 ± 0.10 3.94 ± 0.11 Groundtruth 4.04 ± 0.08 4.05 ± 0.10 Table 7.”
• YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone (2021, background): “, 2018), allowing for speech generation that resembles the target speaker (Jia et al., 2018).”
• Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance (2021, methodology): “There have also been approaches using untranscribed speech to adapt the pre-trained multi-speaker TTS model for fewshot TTS synthesis (Jia et al., 2018; Yan et al., 2021).”
• Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech (2021, methodology): “SV2TTS [6] uses the speaker encoding method, where the speaker encoder is transferred from a pre-trained speaker verification model, and the TTS architecture is a Tacotron 2 [36].”
• A Survey on Neural Speech Synthesis (2021, methodology): “04 SV-Tacotron [139] AM ch/ph AR −→melS AR −→wav NeurIPS18 2018.”
• Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, neutral): “To verify that our model can learn and express diverse speech characteristics, we compared our model to Tacotron 2, Glow-TTS and HiFi-GAN, which showed the ability to extend to multi-speaker speech synthesis (Jia et al., 2018; Kim et al., 2020; Kong et al., 2020).”
• AdaSpeech: Adaptive Text to Speech for Custom Voice (2021, methodology): “Previous works use zero-shot adaptation with speaker encoder (Arik et al., 2018; Jia et al., 2018; Cooper et al., 2020) or only fine-tune the speaker embedding cannot achieve satisfied quality.”
• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech (2019, methodology): “Although this corpus was originally designed for automaticspeechrecognition (ASR)research, ithasbeenusedin various text-to-speech (TTS) research projects [7,8,11] thanks to its attractive properties, such as a non-restrictive license, a large amount of data, and wide speaker diversity.”
• Sample Efficient Adaptive Text-to-Speech (2018, methodology): “As a rough comparisson, because of varying training setups, the state of the art system of Jia et al. (2018) achieves scores of 3.03 for LibriSpeech and 2.77 for VCTK when trained on LibriSpeech.”