Voice Synthesis for in-the-Wild Speakers via a Phonological Loop

By Yaniv Taigman, Lior Wolf, Adam Polyak, Eliya Nachmani. 2017

28 citations recorded by the source corpus.

We present a new neural text to speech method that is able to transform text to speech in voices that are sampled in the wild. Unlike other text to speech systems, our solution is able to deal with unconstrained samples obtained from public speeches. The network architecture is simpler than those in the existing literature and is based on a novel shifting buffer working memory. The same buffer is used for estimating the attention, computing the output audio, and for updating the buffer itself. The input sentence is encoded using a context-free lookup table that contains one entry per character or phoneme. Lastly, the speakers are similarly represented by a short vector that can also be fitted to new speakers and variability in the generated speech is achieved by priming the buffer prior to generating the audio. Experimental results on two datasets demonstrate convincing multi-speaker and in-the-wild capabilities. In order to promote reproducibility, we release our source code and models: PyTorch code and sample audio files are available at ytaigman.github.io/loop.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tie-Yan Liu
3. Tao Qin
4. Yu Zhang
5. Yonghui Wu
6. Ye Jia
7. Lei He
8. Ron J. Weiss

In Speech & Audio

1. Yu Zhang
2. Xu Tan
3. Yonghui Wu
4. Ye Jia
5. Tao Qin
6. Ron J. Weiss
7. Lei He
8. Heiga Zen

In Signal Processing

1. Yu Zhang
2. Heiga Zen
3. Yonghui Wu
4. Ron J. Weiss
5. Xu Tan
6. Ye Jia
7. Junichi Yamagishi
8. Lei He

Citations in the corpus

• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech (2019, background): “Thus, the focus ofTTS research is shifting towardmore challenging tasks, such asmulti-speaker TTS [6–8], building TTS systems from small amounts of data [9], few shot voice adaptation [8, 10, 11], unsupervised prosody and speaking style modeling [12, 13], and the use of found data [13, 14].”
• Neural Voice Cloning with a Few Samples (2018, methodology): “n is to fine-tune a trained multispeaker model for an unseen speaker using a few audios and corresponding texts by applying gradient descent. Finetuning can be applied to either the speaker embedding (Taigman et al., 2017) or the whole model. For embedding-only adaptation, we have the following objec”
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis (2018, background): “The past few years have seen exciting developments in the use of deep neural networks to synthesize natural-sounding human speech (Zen et al., 2016; van den Oord et al., 2016; Wang et al., 2017a; Arik et al., 2017; Taigman et al., 2017; Shen et al., 2017).”
• Deep Voice 3: 2000-Speaker Neural Text-to-Speech (2017, background): “…the problem of synthesizing speech with neural networks, including Deep Voice 1 (Arık et al., 2017), Deep Voice 2 (Arık et al., 2017), Tacotron (Wang et al., 2017), Char2Wav (Sotelo et al., 2017), VoiceLoop (Taigman et al., 2017), SampleRNN (Mehri et al., 2017), and WaveNet (Oord et al., 2016).”
• Uncovering Latent Style Factors for Expressive Speech Synthesis (2017, unknown): “While it’s possible to produce multiple prosodies by other methods (such as priming the decoder initial states, as done in [6]), using attention-based style tokens has several benefits.”
• CSS10: A Collection of Single Speaker Speech Datasets for 10 Languages (2019, unknown): “Among them are WaveNet [1], Tacotron 1 & 2 [2][3], Char2Wav [4], DeepVoice 1, 2, & 3 [5][6][7], DCTTS [8], VoiceLoop [9], etc.”
• Representation Mixing for TTS Synthesis (2018, unknown): “We use a Gaussian mixture attention, first described by Graves [19] and used in several speech related papers [3, 29, 30], though we utilize soft-plus activation for the mean-step parameter finding that it improved stability in early training.”
• Predicting Expressive Speaking Style from Text in End-To-End Speech Synthesis (2018, background): “The recently-published VAE-Loop [18] aims to learn speaking style variations by conditioning VoiceLoop [19], an autoregressive speech synthesis model, on the global latent variable output by a conditional variational autoencoder (VAE).”
• Emotional End-to-End Neural Speech Synthesizer (2017, methodology): “A stack of RNNs is used for the decoder together with a content-based tanh attention mechanism [8] in the original Tacotron.”
• SING: Symbol-to-Instrument Neural Generator (2018, background): “This represents a shift of paradigm with respect to approaches that generate sequences of parameters to vocoders in text-to-speech systems [21, 23, 19], or MIDI partition in music generation [8, 3, 10].”
• Normalization Driven Zero-Shot Multi-Speaker Speech Synthesis (2021, background): “Some of the current approaches [6, 7, 8, 9, 10, 11, 12] have used speaker embedding to capture the identity and speaking style of the person in the speech.”
• Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning (2017, background): “…the problem of synthesizing speech with neural networks, including Deep Voice 1 (Arık et al., 2017), Deep Voice 2 (Arık et al., 2017), Tacotron (Wang et al., 2017), Char2Wav (Sotelo et al., 2017), VoiceLoop (Taigman et al., 2017), SampleRNN (Mehri et al., 2017), and WaveNet (Oord et al., 2016).”