VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature

By Chenpeng Du, Yiwei Guo, Xie Chen, K. Yu. 2022

68 citations recorded by the source corpus.

The mainstream neural text-to-speech(TTS) pipeline is a cascade system, including an acoustic model(AM) that predicts acoustic feature from the input transcript and a vocoder that generates waveform according to the given acoustic feature. However, the acoustic feature in current TTS systems is typically mel-spectrogram, which is highly correlated along both time and frequency axes in a complicated way, leading to a great difficulty for the AM to predict. Although high-fidelity audio can be generated by recent neural vocoders from ground-truth(GT) mel-spectrogram, the gap between the GT and the predicted mel-spectrogram from AM degrades the performance of the entire TTS system. In this work, we propose VQTTS, consisting of an AM txt2vec and a vocoder vec2wav, which uses self-supervised vector-quantized(VQ) acoustic feature rather than mel-spectrogram. We redesign both the AM and the vocoder accordingly. In particular, txt2vec basically becomes a classification model instead of a traditional regression model while vec2wav uses an additional feature encoder before HifiGAN generator for smoothing the discontinuous quantized feature. Our experiments show that vec2wav achieves better reconstruction performance than HifiGAN when using self-supervised VQ acoustic feature. Moreover, our entire TTS system VQTTS achieves state-of-the-art performance in terms of naturalness among all current publicly available TTS systems.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Dong Yu
3. Zhuo Chen
4. Shujie Liu
5. Nima Mesgarani
6. Lei He
7. Yu Wu
8. Sanyuan Chen

In Speech & Audio

1. Jinyu Li
2. Lei He
3. Zhuo Chen
4. Dong Yu
5. Sheng Zhao
6. Rongjie Huang
7. Shujie Liu
8. Nima Mesgarani

In Signal Processing

1. Jinyu Li
2. Dong Yu
3. Zhuo Chen
4. Nima Mesgarani
5. Shujie Liu
6. Lei He
7. Yu Wu
8. Sanyuan Chen

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, positive): “2) There is an off-the-shelf codec decoder to convert discrete tokens into a waveform, without the additional efforts on vocoder training like VQ-based methods that operated on spectrum [Du et al., 2022].”
• UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “Recently, VQTTS (Du et al. 2022) proposes a novel approach by utilizing discrete speech tokens as the intermediate representation for text-to-speech synthesis.”
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, background): “Recent advancements have proven the effectiveness of large-scale self-supervised speech language models (SLMs) in enhancing text-tospeech (TTS) quality [34, 35, 36, 37] and speaker adaptation [8, 38, 29, 39].”
• InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, methodology): “A similar idea modeling mel-spectrogram in discrete latent space is applied in VQ-TTS [55], which utilizes self-supervised VQ acoustic feature (vq-wav2vec [18]) rather than traditional mel-spectrogram as intermediate prediction target.”
• DCTTS: Discrete Diffusion Model with Contrastive Learning for Text-to-Speech Generation (2023, methodology): “This method not only greatly reduces computational consumption, but also enhances the generalization of the TTS model[10].”
• HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “Meanwhile, there are text prompt-based style generation models which can describe a speaking or voice style from text descriptions [61], [62], [63].”
• FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model (2023, methodology): “VQTTS [31], consists of an acoustic model (AM) txt2vec and a vocoder vec2wav, which uses selfsupervised vector-quantized (VQ) acoustic feature rather than mel-spectrogram, in which txt2vec becomes a classification ar X iv :2 30 3.”