UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding
By Chenpeng Du, Yiwei Guo, Feiyu Shen, Zhijun Liu, Zheng Liang, Xie Chen, Shuai Wang, Hui Zhang, K. Yu. 2023
66 citations recorded by the source corpus.
The utilization of discrete speech tokens, divided into semantic tokens and acoustic tokens, has been proven superior to traditional acoustic feature mel-spectrograms in terms of naturalness and robustness for text-to-speech (TTS) synthesis. Recent popular models, such as VALL-E and SPEAR-TTS, allow zero-shot speaker adaptation through auto-regressive (AR) continuation of acoustic tokens extracted from a short speech prompt. However, these AR models are restricted to generate speech only in a left-to-right direction, making them unsuitable for speech editing where both preceding and following contexts are provided. Furthermore, these models rely on acoustic tokens, which have audio quality limitations imposed by the performance of audio codec models. In this study, we propose a unified context-aware TTS framework called UniCATS, which is capable of both speech continuation and editing. UniCATS comprises two components, an acoustic model CTX-txt2vec and a vocoder CTX-vec2wav. CTX-txt2vec employs contextual VQ-diffusion to predict semantic tokens from the input text, enabling it to incorporate the semantic context and maintain seamless concatenation with the surrounding context. Following that, CTX-vec2wav utilizes contextual vocoding to convert these semantic tokens into waveforms, taking into consideration the acoustic context. Our experimental results demonstrate that CTX-vec2wav outperforms HifiGAN and AudioLM in terms of speech resynthesis from semantic tokens. Moreover, we show that UniCATS achieves state-of-the-art performance in both speech continuation and editing. Audio samples are available at https://cpdu.github.io/unicats.
Subfields: Artificial Intelligence, Generative Models, Speech & Audio
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Generative Models
In Speech & Audio
Citations in the corpus
- Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “StyleTTS 2 [53], UniCATS [54], NaturalSpeech 2 [55] and NaturalSpeech 3 [56] employed diffusion model [57] for the prompt-conditioned text to speech synthesis.”
- VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “StyleTTS 2 [Li et al., 2024], UniCATS [Du et al., 2024a], NaturalSpeech 2 [Shen et al., 2023] and NaturalSpeech 3 [Ju et al., 2024] used diffusion model [Ho et al., 2020] for the prompt-conditioned text to speech synthesis.”
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “…these systems into four groups based on data representation and modelling methods: 1) Discrete Tokens + Autoregressive [6, 19, 20]; 2) Discrete Tokens + Non-2 autoregressive [13, 21, 22]; 3) Continuous Vectors + Autoregressive [23]; 4) Continuous Vectors + Non-autoregressive [5, 11, 24, 25].”
- Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS (2023, methodology): “Subsequently, Du et al. [15] introduced UniCATS, a TTS system based on semantic tokens and free of acoustic tokens and speaker embeddings.”
- CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models (2024, unknown): “The first one is constructed from the test-clean set of Librispeech corpus [51], denoting as test-clean .”
- Seed-TTS: A Family of High-Quality Versatile Speech Generation Models (2024, methodology): “…applied methods to reduce the memory and computation consumption on the language model side, such as grouped-query attention [Ainslie et al., 2023], paged attention [Kwon et al., 2023], flash attention [Dao et al., 2022, Dao, 2023], and model quantization [Nagel et al., 2021, Guo et al., 2024].”