Neural Voice Cloning with a Few Samples

By Sercan Ö. Arik, Jitong Chen, Kainan Peng, Wei Ping, Yanqi Zhou. 2018

443 citations recorded by the source corpus.

Voice cloning is a highly desired feature for personalized speech interfaces. Neural network based speech synthesis has been shown to generate high quality speech for a large number of speakers. In this paper, we introduce a neural voice cloning system that takes a few audio samples as input. We study two approaches: speaker adaptation and speaker encoding. Speaker adaptation is based on fine-tuning a multi-speaker generative model with a few cloning samples. Speaker encoding is based on training a separate model to directly infer a new speaker embedding from cloning audios and to be used with a multi-speaker generative model. In terms of naturalness of the speech and its similarity to original speaker, both approaches can achieve good performance, even with very few cloning audios. While speaker adaptation can achieve better naturalness and similarity, the cloning time or required memory for the speaker encoding approach is significantly less, making it favorable for low-resource deployment.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, positive): “To tackle the zero-shot TTS problem, existing work leverages speaker adaptation and speaker encoding methods, requiring additional ﬁne-tuning, complex pre-designed features, or heavy structure engineering [6], [7], [8], [9].”
Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis (2018, methodology): “Improving speaker similarity given more than a few seconds of reference speech requires a model adaptation approach as in [2], and more recently in [5].”
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “Early work in zero-shot TTS typically employed speaker adaptation and speaker encoding methods, which often required additional fine-tuning, complex pre-designed features, or heavy structure engineering [Chen et al., 2019, Wang et al., 2020, Arik et al., 2018, Casanova et al., 2022].”
Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, background): “Previous approaches can be categorized into speaker adaptation [13, 11, 60, 23] and speaker encoding [25, 1, 26, 61] methods.”
Grad-StyleSpeech: Any-Speaker Adaptive Text-to-Speech Synthesis with Diffusion Models (2022, background): “On the other hand, recent works [12, 14, 15] have focused on a zero-shot approach in which transcribed samples and the additional fine-tuning stage are not necessarily required to adapt to the unseen speaker, thanks to their neural encoder that encode any speech into the latent vector.”
YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone (2021, neutral): “ZS-TTS using deep learning was ﬁrst proposed by [5] which extended the DeepVoice 3 method [6].”
Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance (2021, methodology): “Speaker embedding based approach (Arik et al., 2018; Jia et al., 2018; Casanova et al., 2021) provides the target speaker’s embedding extracted from untranscribed speech to the TTS model for adaptation.”
Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech (2021, methodology): “[8] uses DeepVoice3 [5] as the base model architecture.”
A Survey on Neural Speech Synthesis (2021, background): “It is known as different terms in academia and industry, such as voice adaptation [43], voice cloning [9], custom voice [39], etc.”
AdaSpeech: Adaptive Text to Speech for Custom Voice (2021, methodology): “…a large amount of high-quality and single-speaker recordings (Ito, 2017), and has been extended to multi-speaker scenarios (Gibiansky et al., 2017; Ping et al., 2018; Zen et al., 2019; Chen et al., 2020) using multi-speaker corpora (Panayotov et al., 2015; Veaux et al., 2016; Zen et al., 2019).”
LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech (2019, background): “Thus, the focus ofTTS research is shifting towardmore challenging tasks, such asmulti-speaker TTS [6–8], building TTS systems from small amounts of data [9], few shot voice adaptation [8, 10, 11], unsupervised prosody and speaking style modeling [12, 13], and the use of found data [13, 14].”
Sample Efficient Adaptive Text-to-Speech (2018, methodology): “The most similar approached to our work was proposed by Arik et al. (2018) for the DeepVoice 3 model.”
Hierarchical Generative Modeling for Controllable Speech Synthesis (2018, background): “The proposed formulation for learning an observed attribute embedding z o for speaker modeling is also related to Arik et al. (2018), which controls the speaker identity with speaker embeddings, and trains a separate regression model to predict them from the audio.”
Direct speech-to-speech translation with a sequence-to-sequence model (2019, background): “Similarly, recent work on neural TTS has focused on adapting to new voices with limited reference data [13–16].”
BigVGAN: A Universal Neural Vocoder with Large-Scale Training (2022, methodology): “speakers [39], neural voice cloning [2, 13], voice conversion [26], speech-to-speech translation [12], and neural audio codec [59], where the studio-quality recordings are often not available.”
Zero-Shot Multi-Speaker Text-To-Speech with State-Of-The-Art Neural Speaker Embeddings (2019, methodology): “Prior studies have focused on training a speaker encoder network jointly with the TTS model [6, 8, 9] or the neural vocoder [10]; others have explored the use of speaker embeddings in combination with fine-tuning the TTS model [10, 11, 12].”
Fast Spectrogram Inversion Using Multi-Head Convolutional Neural Networks (2018, methodology): “2Human ratings are collected via Amazon Mechanical Turk framework independently for each evaluation, as in [19].”
Almost Unsupervised Text to Speech and Automatic Speech Recognition (2019, background): “…Chen et al., 2018b), low-resource ASR (Chuang-suwanich, 2016; Dalmia et al., 2018; Zhou et al., 2018), TTS with minimal speaker data (Chen et al., 2019; Jia et al., 2018; Arik et al., 2018; Wang et al., 2018), and boosting TTS and ASR simultaneously in a speech chain (Tjandra et al., 2017; 2018).”
Generalizing from a Few Examples (2019, background): “Learning is stopped when there is no performance improvement on the validation set [6].”