WavThruVec: Latent speech representation as intermediate features for neural speech synthesis

By Hubert Siuzdak, Piotr Dura, Pol van Rijn, Nori Jacoby. 2022

39 citations recorded by the source corpus.

Recent advances in neural text-to-speech research have been dominated by two-stage pipelines utilizing low-level intermediate speech representation such as mel-spectrograms. However, such predetermined features are fundamentally limited, because they do not allow to exploit the full potential of a data-driven approach through learning hidden representations. For this rea-son, several end-to-end methods have been proposed. However, such models are harder to train and require a large number of high-quality recordings with transcriptions. Here, we propose WavThruVec – a two-stage architecture that resolves the bot-tleneck by using high-dimensional WAV 2 VEC 2.0 embeddings as intermediate speech representation. Since these hidden activations provide high-level linguistic features, they are more robust to noise. That allows us to utilize annotated speech datasets of a lower quality to train the ﬁrst-stage module. At the same time, the second-stage component can be trained on large-scale untranscribed audio corpora, as WAV 2 VEC 2.0 embeddings are already time-aligned. This results in an increased generalization capability to out-of-vocabulary words, as well as to a better generalization to unseen speakers. We show that the proposed model not only matches the quality of state-of-the-art neural models, but also presents useful properties enabling tasks like voice conversion or zero-shot synthesis.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, background): “Recent advancements have proven the effectiveness of large-scale self-supervised speech language models (SLMs) in enhancing text-tospeech (TTS) quality [34, 35, 36, 37] and speaker adaptation [8, 38, 29, 39].”
Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, background): “Furthermore, recent research is focusing on leveraging latent features extracted by pretrained deep learning models (Polyak et al., 2021; Siuzdak et al., 2022).”
HierSpeech: Bridging the Gap between Text and Speech by Hierarchical Variational Inference using Self-supervised Representations for Speech Synthesis (2022, unknown): “Recently, self-supervised speech representations have been utilized in TTS tasks (Siuzdak et al., 2022; Kim et al., 2022; Du et al., 2022) and we see that our hierarchical speech synthesis structure using self-supervised speech representations can be also utilized for various tasks.”
NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis (2022, methodology): “For TTS applications, Siuzdak et al. (2022) proposed to employ wav2vec 2.0 feature as a mid-level representation of a TTS model.”
HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “Like AMS-SFE [25] aligning manifold structures via semantic features, ZS-TTS [26], WavthruVec [27], and VQTTS [28] used SSL representation as an intermediate acoustic representation for robust speech synthesis.”