NAUTILUS: A Versatile Voice Cloning System

By Hieu-Thi Luong, Junichi Yamagishi. 2020

61 citations recorded by the source corpus.

We introduce a novel speech synthesis system, called NAUTILUS, that can generate speech with a target voice either from a text input or a reference utterance of an arbitrary source speaker. By using a multi-speaker speech corpus to train all requisite encoders and decoders in the initial training stage, our system can clone unseen voices using untranscribed speech of target speakers on the basis of the backpropagation algorithm. Moreover, depending on the data circumstance of the target speaker, the cloning strategy can be adjusted to take advantage of additional data and modify the behaviors of text-to-speech (TTS) and/or voice conversion (VC) systems to accommodate the situation. We test the performance of the proposed framework by using deep convolution layers to model the encoders, decoders and WaveNet vocoder. Evaluations show that it achieves comparable quality with state-of-the-art TTS and VC systems when cloning with just five minutes of untranscribed speech. Moreover, it is demonstrated that the proposed framework has the ability to switch between TTS and VC with high speaker consistency, which will be useful for many applications.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Junichi Yamagishi
6. Sheng Zhao
7. Haizhou Li
8. Berrak Sisman

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Junichi Yamagishi
6. Sheng Zhao
7. Haizhou Li
8. Berrak Sisman

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, background): “Efficient Adaptation Few-Data Adaptation [43, 9, 173, 234, 439, 48, 39, 230] Untranscribed Data Adaptation [397, 130, 215] Few-Parameter Adaptation [9, 43, 39] Zero-Shot Adaptation [9, 43, 139, 55]”
• Adaspeech 2: Adaptive Text to Speech with Untranscribed Data (2021, methodology): “The alignment process above distinguishes our work from previous works [11] that also adopt hidden sequence alignment to align the phoneme and mel encoder.”
• An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning (2020, methodology): “Recent studies introduced a framework for creating limited-data VC system [222], [224], [225] by bootstrapping from a speaker-adaptive TTS model.”
• Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, methodology): “The key distinction between voice cloning and speech synthesis systems is that the former focuses on preserving the characteristics of the specific identity speech attributes while the latter lacks this feature to maintain the quality of the generated speech [190].”
• Pretraining Techniques for Sequence-to-Sequence Voice Conversion (2020, background): “can be categorized into the speech synthesis task [29], so the respective models should share some characteristics.”