NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis

By Hyeong-Seok Choi, Jinhyeok Yang, Juheon Lee, Hyeongju Kim. 2022

60 citations recorded by the source corpus.

Various applications of voice synthesis have been developed independently despite the fact that they generate"voice"as output in common. In addition, most of the voice synthesis models still require a large number of audio data paired with annotated labels (e.g., text transcription and music score) for training. To this end, we propose a unified framework of synthesizing and manipulating voice signals from analysis features, dubbed NANSY++. The backbone network of NANSY++ is trained in a self-supervised manner that does not require any annotations paired with audio. After training the backbone network, we efficiently tackle four voice applications - i.e. voice conversion, text-to-speech, singing voice synthesis, and voice designing - by partially modeling the analysis features required for each task. Extensive experiments show that the proposed framework offers competitive advantages such as controllability, data efficiency, and fast training convergence, while providing high quality synthesis. Audio samples: tinyurl.com/8tnsy3uc.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Jinyu Li
3. Dong Yu
4. Zhou Zhao
5. Sheng Zhao
6. Tao Qin
7. Tie-Yan Liu
8. Rongjie Huang

In Signal Processing

1. Jinyu Li
2. Dong Yu
3. Xu Tan
4. Zhou Zhao
5. Rongjie Huang
6. Sheng Zhao
7. Chao Weng
8. Dongchao Yang

Citations in the corpus

• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “Prior works [36, 37, 38] utilize disentangled representation for speech generation, such as speech content from self-supervised pre-trained models [39, 40, 41], fundamental frequency, and timbre, but speech quality is not satisfying.”
• Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, methodology): “The limited amount of labeled data [7] and unpleasant glitches [13] are two major challenges for singing voice synthesis, which we address as follows: 1) For data scarcity, Make-A-Voice leverages a large amount of unlabeled speech and singing voice data for modeling the acoustic diversity (speaker,…”
• HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “Meanwhile, there are text prompt-based style generation models which can describe a speaking or voice style from text descriptions [61], [62], [63].”