Make-A-Voice: Unified Voice Synthesis With Discrete Representation

By Rongjie Huang, Chunlei Zhang, Yongqiang Wang, Dongchao Yang, Lu Liu, Zhenhui Ye, Ziyue Jiang, Chao Weng, Zhou Zhao, Dong Yu. 2023

38 citations recorded by the source corpus.

Various applications of voice synthesis have been developed independently despite the fact that they generate"voice"as output in common. In addition, the majority of voice synthesis models currently rely on annotated audio data, but it is crucial to scale them to self-supervised datasets in order to effectively capture the wide range of acoustic variations present in human voice, including speaker identity, emotion, and prosody. In this work, we propose Make-A-Voice, a unified framework for synthesizing and manipulating voice signals from discrete representations. Make-A-Voice leverages a"coarse-to-fine"approach to model the human voice, which involves three stages: 1) semantic stage: model high-level transformation between linguistic content and self-supervised semantic tokens, 2) acoustic stage: introduce varying control signals as acoustic conditions for semantic-to-acoustic modeling, and 3) generation stage: synthesize high-fidelity waveforms from acoustic tokens. Make-A-Voice offers notable benefits as a unified voice synthesis framework: 1) Data scalability: the major backbone (i.e., acoustic and generation stage) does not require any annotations, and thus the training data could be scaled up. 2) Controllability and conditioning flexibility: we investigate different conditioning mechanisms and effectively handle three voice synthesis applications, including text-to-speech (TTS), voice conversion (VC), and singing voice synthesis (SVS) by re-synthesizing the discrete voice representations with prompt guidance. Experimental results demonstrate that Make-A-Voice exhibits superior audio quality and style similarity compared with competitive baseline models. Audio samples are available at https://Make-A-Voice.github.io

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Jinyu Li
3. Shujie Liu
4. Sheng Zhao
5. Zhuo Chen
6. Tao Qin
7. Tie-Yan Liu
8. Yanqing Liu

In Speech & Audio

1. Xu Tan
2. Sheng Zhao
3. Rongjie Huang
4. Jinyu Li
5. Yanqing Liu
6. Shujie Liu
7. Dongchao Yang
8. Zhuo Chen

In Signal Processing

1. Jinyu Li
2. Xu Tan
3. Zhuo Chen
4. Shujie Liu
5. Sheng Zhao
6. Rongjie Huang
7. Sanyuan Chen
8. Takuya Yoshioka

Citations in the corpus

• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “SPEAR-TTS [Kharitonov et al., 2023] and Make-a-voice [Huang et al., 2023a] leveraged semantic units from a speech self-supervised model as an intermediate interface between text and acoustic codec codes, enabling better training data efficiency.”
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “…categorize these systems into four groups based on data representation and modelling methods: 1) Discrete Tokens + Autoregressive [6, 19, 20]; 2) Discrete Tokens + Non-2 autoregressive [13, 21, 22]; 3) Continuous Vectors + Autoregressive [23]; 4) Continuous Vectors + Non-autoregressive…”
• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, methodology): “However, LMs’ ability to process multiple tasks is less exploited in audio generation research: most existing LM-based works are still designed for single tasks (Wang et al., 2023a; Kharitonov et al., 2023).”
• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, methodology): “Consequently, these models have not only achieved remarkable zeroshot TTS performance but also demonstrated additional capabilities, including voice conversion [15], [18] and speech edit-”
• HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “Although all the above models have shown powerful adaptation performance, they have a slow inference speed for their iterative generation manner.”