Sample Efficient Adaptive Text-to-Speech

By Yutian Chen, Yannis Assael, Brendan Shillingford, David M. Budden, Scott E. Reed, Heiga Zen, Quan Wang, Luis C. Cobo, Andrew Trask, Ben Laurie and 4 more. 2018

162 citations recorded by the source corpus.

We present a meta-learning approach for adaptive text-to-speech (TTS) with few data. During training, we learn a multi-speaker model using a shared conditional WaveNet core and independent learned embeddings for each speaker. The aim of training is not to produce a neural network with fixed weights, which is then deployed as a TTS system. Instead, the aim is to produce a network that requires few data at deployment time to rapidly adapt to new speakers. We introduce and benchmark three strategies: (i) learning the speaker embedding while keeping the WaveNet core fixed, (ii) fine-tuning the entire architecture with stochastic gradient descent, and (iii) predicting the speaker embedding with a trained neural network encoder. The experiments show that these approaches are successful at adapting the multi-speaker neural network to new speakers, obtaining state-of-the-art results in both sample naturalness and voice similarity with merely a few minutes of audio data from new speakers.

Subfields: Artificial Intelligence, Efficiency & Compression, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

In Speech & Audio

In Signal Processing

Citations in the corpus

Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis (2018, methodology): “Improving speaker similarity given more than a few seconds of reference speech requires a model adaptation approach as in [2], and more recently in [5].”
Weakly Supervised Disentanglement with Guarantees (2019, methodology): “…disentanglement research has focused on the learning of such representations in an unsupervised fashion, using only independent samples from the data distribution without access to the true factors of variation (Higgins et al., 2017; Chen et al., 2018a; Kim & Mnih, 2018; Esmaeili et al., 2018).”
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “Early work in zero-shot TTS typically employed speaker adaptation and speaker encoding methods, which often required additional fine-tuning, complex pre-designed features, or heavy structure engineering [Chen et al., 2019, Wang et al., 2020, Arik et al., 2018, Casanova et al., 2022].”
Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, background): “Previous approaches can be categorized into speaker adaptation [13, 11, 60, 23] and speaker encoding [25, 1, 26, 61] methods.”
Grad-StyleSpeech: Any-Speaker Adaptive Text-to-Speech Synthesis with Diffusion Models (2022, background): “Some of previous works [13, 16] need a few transcribed (supervised) samples for fine-tuning TTS model.”
Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech (2021, methodology): “First, SEA-TTS applies early stopping by splitting the fewshot support data into train and validation sets.”
A Survey on Neural Speech Synthesis (2021, methodology): “It is known as different terms in academia and industry, such as voice adaptation [43], voice cloning [9], custom voice [39], etc.”
AdaSpeech: Adaptive Text to Speech for Custom Voice (2021, background): “They ﬁne-tune the whole model (Chen et al., 2018; Kons et al., 2019) or decoder part (Moss et al., 2020; Zhang et al., 2020), achieving good quality but causing too many adaptation parameters.”
LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech (2019, methodology): “Although this corpus was originally designed for automaticspeechrecognition (ASR)research, ithasbeenusedin various text-to-speech (TTS) research projects [7,8,11] thanks to its attractive properties, such as a non-restrictive license, a large amount of data, and wide speaker diversity.”
Low Bit-rate Speech Coding with VQ-VAE and a WaveNet Decoder (2019, methodology): “We follow Jia et al. [19] and Chen et al. [20] who assess speaker similarity using mean opinion score (MOS) evaluations based on subjective listening tests.”
Direct speech-to-speech translation with a sequence-to-sequence model (2019, background): “[12] transfered source speaker’s voice to the synthesized translated speech by mapping hidden Markov model states from ASR to TTS. Similarly, recent work on neural TTS has focused on adapting to new voices with limited reference data [13–16].”
Zero-Shot Multi-Speaker Text-To-Speech with State-Of-The-Art Neural Speaker Embeddings (2019, methodology): “An effective approach for speaker adaptation in neural TTS is to ﬁne-tune all or part of model with a small amount of data from the target speaker [5, 6].”
MelNet: A Generative Model for Audio in the Frequency Domain (2019, methodology): “Notable such examples include the application of WaveNet to the tasks of text-tospeech [47, 50, 26, 6] and MIDI-conditional music generation [19, 33].”
Almost Unsupervised Text to Speech and Automatic Speech Recognition (2019, background): “…et al., 2018; Chen et al., 2018b), low-resource ASR (Chuang-suwanich, 2016; Dalmia et al., 2018; Zhou et al., 2018), TTS with minimal speaker data (Chen et al., 2019; Jia et al., 2018; Arik et al., 2018; Wang et al., 2018), and boosting TTS and ASR simultaneously in a speech chain (Tjandra et…”
Adaspeech 2: Adaptive Text to Speech with Untranscribed Data (2021, background): “Most previous works [1, 3, 4, 8] on TTS adaptation require paired data (speech and its transcripts), such as AdaSpeech [8].”
Meta-learning of Sequential Strategies (2019, methodology): “…learning process, such as learning the optimizer [Andrychowicz et al., 2016, Li and Malik, 2016, Ravi and Larochelle, 2016, Wichrowska et al., 2017, Chen et al., 2018b], the metric space [Vinyals et al., 2016, Snell et al., 2017], the initial network parameters [Finn et al., 2017, Nichol and…”