Zero-Shot Multi-Speaker Text-To-Speech with State-Of-The-Art Neural Speaker Embeddings

By Erica Cooper, Cheng-I Lai, Yusuke Yasuda, Fuming Fang, Xin Eric Wang, Nanxin Chen, Junichi Yamagishi. 2019

204 citations recorded by the source corpus.

While speaker adaptation for end-to-end speech synthesis using speaker embeddings can produce good speaker similarity for speakers seen during training, there remains a gap for zero-shot adaptation to unseen speakers. We investigate multi-speaker modeling for end-to-end text-to-speech synthesis and study the effects of different types of state-of-the-art neural speaker embeddings on speaker similarity for unseen speakers. Learnable dictionary encoding-based speaker embeddings with angular softmax loss can improve equal error rates over x-vectors in a speaker verification task; these embeddings also improve speaker similarity and naturalness for unseen speakers when used for zero-shot adaptation to new speakers in end-to-end speech synthesis.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, background): “Early studies in zero-shot TTS employed fixeddimensional speaker embeddings [11], [12], [13], [14].”
PolyVoice: Language Models for Speech to Speech Translation (2023, background): “Because of the requirements of real-world applications, the researchers have attracted a lot of attention to zero-shot multi-speaker TTS and crosslingual TTS (Jia et al., 2018; Cooper et al., 2020).”
AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios (2022, background): “For the first step, previous zero-shot TTS models [16, 17, 18] usually leverage a speaker/reference encoder to extract speaker representation of a target speaker.”
YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone (2021, methodology): “It ranges from -1 to 1, and a larger value indicates a stronger similarity (Cooper et al., 2020).”
Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech (2021, background): “There are two general approaches to deal with such task [8], speaker adaptation [8], [9], [10], [11], [12] and speaker encoding [6], [13], [7], [14], [15], [16], [17].”
A Survey on Neural Speech Synthesis (2021, background): “Some works [9, 43, 139, 55, 32] conduct zero-shot adaptation, which leverage a speaker encoder to extract speaker embeddings given a reference audio.”
AdaSpeech: Adaptive Text to Speech for Custom Voice (2021, methodology): “Previous works use zero-shot adaptation with speaker encoder (Arik et al., 2018; Jia et al., 2018; Cooper et al., 2020) or only ﬁne-tune the speaker embedding cannot achieve satisﬁed quality.”
Adaspeech 2: Adaptive Text to Speech with Untranscribed Data (2021, background): “TTS adaptation pipeline [1, 2, 3, 4, 5, 6, 7, 8] usually finetunes a well-trained multi-speaker TTS model [9, 10] (source model) with few adaptation data, and achieves good similarity with ground-truth human voice and high voice quality.”
SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model (2021, methodology): “Following the proposal of [2], [21] and [3] we use local sensitive attention [6].”
Improved Prosody from Learned F0 Codebook Representations for VQ-VAE Speech Waveform Reconstruction (2020, background): “For more details, refer to Section 2 of [16] and its implementation 3 .”
XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model (2024, unknown): “Meanwhile, Tacotron 2 [9] was adapted using external speaker embeddings, allowing for speech generation that resembles the target speaker [1, 10].”
Can Speaker Augmentation Improve Multi-Speaker End-to-End TTS? (2020, methodology): “For more details, refer to Section 2 of [9].”
Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding (2020, unknown): “We refer to our proposed model as Attentron and base-lines, which can synthesize speech from speakers unseen during training as LDE [13] and GMVAE [17].”
The AS-NU System for the M2VoC Challenge (2021, background): “Such a challenging task is referred to as voice cloning, speaker adaptation or few-shot/one-shot speech synthesis [7, 8, 9, 10, 11, 12].”
Mega-TTS 2: Zero-Shot Text-to-Speech with Arbitrary Length Speech Prompts (2023, background): “With the advancement in this field, advanced TTS systems are capable of synthesizing high-quality voices of single or multiple speakers (Jia et al., 2018; Cooper et al., 2020; Chen et al., 2021b; Casanova et al., 2022; Huang et al., 2022; Kim et al., 2022b; Kang et al., 2023).”
Pretraining Strategies, Waveform Model Choice, and Acoustic Configurations for Multi-Speaker End-to-End Speech Synthesis (2020, methodology): “For multi-speaker text-to-speech synthesis, [7] found that warmstarting the multi-speaker model from a high-quality singlespeaker pretrained model can greatly reduce training time as compared to training a multi-speaker model from scratch, and that the warm-starting approach also provides the benefi”
From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraint (2020, methodology): “published a study investigating how different speaker embedding networks affects the multispeaker synthesis system [22].”
Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “[41] extracted speaker information by using learnable dictionary encoding (LDE) on the basis of Jia et al.”
ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wild (2022, background): “TTS and VC technology has advanced rapidly in recent years, especially in terms of requiring far less target speaker training data [53]– [55].”