Joint training framework for text-to-speech and voice conversion using multi-source Tacotron and WaveNet

By Mingyang Zhang, Xin Eric Wang, Fuming Fang, Haizhou Li, Junichi Yamagishi. 2019

59 citations recorded by the source corpus.

We investigated the training of a shared model for both text-to-speech (TTS) and voice conversion (VC) tasks. We propose using an extended model architecture of Tacotron, that is a multi-source sequence-to-sequence model with a dual attention mechanism as the shared model for both the TTS and VC tasks. This model can accomplish these two different tasks respectively according to the type of input. An end-to-end speech synthesis task is conducted when the model is given text as the input while a sequence-to-sequence voice conversion task is conducted when it is given the speech of a source speaker as the input. Waveform signals are generated by using WaveNet, which is conditioned by using a predicted mel-spectrogram. We propose jointly training a shared model as a decoder for a target speaker that supports multiple sources. Listening experiments show that our proposed multi-source encoder-decoder model can efficiently achieve both the TTS and VC tasks.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Junichi Yamagishi
5. Frank K. Soong
6. Haizhou Li
7. Hung-yi Lee
8. Berrak Sisman

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Junichi Yamagishi
5. Frank K. Soong
6. Haizhou Li
7. Berrak Sisman
8. Simon King

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “Self-supervised pretraining methods can be leveraged to enhance the language understanding or speech generation capabilities [51, 368, 433, 78].”
• Many-to-Many Voice Transformer Network (2020, background): “proposed a shared model for TTS and VC tasks that allows for joint training of the TTS and VC functions [42].”
• An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning (2020, methodology): “Tacotron [89], [180], [218]–[220] represents one of the successful text-to-speech (TTS) implementations, that has been extended to voice conversion [3], [183], [221].”
• Pretraining Techniques for Sequence-to-Sequence Voice Conversion (2020, background): “Another line of work show that training an integrated system capable of performing either TTS or VC can boost individual performances [29], [54].”
• NoiseVC: Towards High Quality Zero-Shot Voice Conversion (2021, unknown): “In [17, 18], a Text-to-Speech model is introduced to supervise the training of latent representations of a VC system.”
• SpeechNet: A Universal Modularized Model for Speech Processing Tasks (2021, methodology): “Some papers (Jain et al., 2018; Chen et al., 2020) use SC to improve ASR, and (Tang et al., 2016) studies MTL of ASR and SC. (Hsu et al., 2019; Jia et al., 2018) shows the effectiveness of SC to help TTS. (Zhang et al., 2019b) shows the performance of VC can be improved with text supervision.”
• Emotional Voice Conversion: Theory, Databases and ESD (2021, methodology): “…et al., 2014; Lorenzo-Trueba et al., 2018a), recurrent neural network (RNN) (Nakashika et al., 2014), generative adversarial network (GAN) (Sisman et al., 2018c) and sequence-to-sequence model with attention mechanism (Zhang et al., 2019b; Huang et al., 2019) have advanced the state-of-the-art.”