Fitting New Speakers Based on a Short Untranscribed Sample

By Eliya Nachmani, Adam Polyak, Yaniv Taigman, Lior Wolf. 2018

88 citations recorded by the source corpus.

Learning-based Text To Speech systems have the potential to generalize from one speaker to the next and thus require a relatively short sample of any new voice. However, this promise is currently largely unrealized. We present a method that is designed to capture a new speaker from a short untranscribed audio sample. This is done by employing an additional network that given an audio sample, places the speaker in the embedding space. This network is trained as part of the speech synthesis system using various consistency losses. Our results demonstrate a greatly improved performance on both the dataset speakers, and, more importantly, when fitting new voices, even from very short samples.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis (2018, methodology): “In contrast to [10], we don’t introduce additional loss terms based on the speaker embedding.”
A Survey on Neural Speech Synthesis (2021, background): “Other works [238, 202, 30, 299, 393, 6] leverage the feedback loss to ensure the controllability on style and speaker embeddings, etc.”
LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech (2019, methodology): “The VCTK corpus [20] is popular for experimenting with multi-speaker TTS [6, 8, 11, 14, 29–31] as it contains studio quality speech data from multiple speakers.”
Hierarchical Generative Modeling for Controllable Speech Synthesis (2018, neutral): “Previous results on similar data focused on speaker modeling (Ping et al., 2018; Nachmani et al., 2018; Arik et al., 2018; Jia et al., 2018), and did not explicitly address modeling of prosody and background noise.”
Direct speech-to-speech translation with a sequence-to-sequence model (2019, background): “Potential strategies to improve voice transfer performance include improving the speaker encoder by adding a language adversarial loss, or by incorporating a cycle-consistency term [13] into the S2ST loss.”
Zero-Shot Multi-Speaker Text-To-Speech with State-Of-The-Art Neural Speaker Embeddings (2019, methodology): “Several recent studies [9, 13, 14, 15] have used this approach for speaker modeling in TTS, with [15] modeling both speaker and language characteristics.”
Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation (2021, background): “Following Nachmani et al. (2018), we expect that the ability to adapt to new speakers may depend on the length of the reference speech audio.”
Unpaired Image-to-Speech Synthesis With Multimodal Information Bottleneck (2019, methodology): “Text-to-speech synthesis using deep neural networks has gained much attention recently, with methods such as WaveNet [44], DeepVoice [4, 13, 34], VoiceLoop [42, 30], Char2Wav [39], and Tacotron [49, 50].”
From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraint (2020, methodology): “When the two models are trained jointly, the TTS system yields moderate performance in synthesizing voices that are unseen in the training data [15].”
Unsupervised Singing Voice Conversion (2019, methodology): “The similarity of the generated voices to the target voices is also evaluated automatically, using the same type of classification network that was used to verify voice identifiability in previous work [30, 31, 32, 33].”
Using Vaes and Normalizing Flows for One-Shot Text-To-Speech Synthesis of Expressive Speech (2019, unknown): “problem is also distinct from that of speaker-adaptation [6, 7], as we wish to improve expressiveness by including emotion for existing speakers, not learn entirely new speakers.”
Non-Autoregressive Neural Text-to-Speech (2019, background): “Their excellent extensibility leads to promising results for several challenging tasks, such as voice cloning (Arik et al., 2018; Nachmani et al., 2018; Jia et al., 2018; Chen et al., 2019).”
Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning (2019, background): “Recent end-to-end neural TTSmodels [1–3] have been extended to enable control of speaker identity [4–7] as well as unlabelled speech attributes, e.”
Bi-Level Speaker Supervision for One-Shot Speech Synthesis (2020, methodology): “In [17], a contrastive triplet loss is used to ensure that speaker embeddings predicted by the identical speaker are closer than the embedding computed from different speakers.”
Disentangled Speaker and Language Representations Using Mutual Information Minimization and Domain Adaptation for Cross-Lingual TTS (2021, background): “Recent end-to-end multilingual and multi-speaker text-to-speech (TTS) synthesis systems typically rely on distributed speaker and language representations to model different speakers and languages [1, 2, 3, 4, 5, 6].”
Unsupervised Learning For Sequence-to-sequence Text-to-speech For Low-resource Languages (2020, background): “Some investigated modeling speaker identities using speaker embeddings in TTS [8, 9].”
Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “[145] also a contrastive loss term and a cyclic loss term, which allow the model to synthesize the voice of a new speaker with only a small amount of audio.”
End-to-End Code-Switching TTS with Cross-Lingual Language Model (2020, background): “Speaker embedding is widely used in multi-speaker TTS [17, 18, 19] by characterizing a speaker with a low dimensional vector.”
Investigation of Fast and Efficient Methods for Multi-Speaker Modeling and Speaker Adaptation (2021, methodology): “The speaker embedding could be obtained either by a speaker encoder jointly trained with the TTS model or a model separately trained for a speaker veriﬁcation task [9, 10, 11].”