Speech Resynthesis from Discrete Disentangled Self-Supervised Representations

By Adam Polyak, Yossi Adi, Jade Copet, Evgeny Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdel-rahman Mohamed, Emmanuel Dupoux. 2021

398 citations recorded by the source corpus.

We propose using self-supervised discrete representations for the task of speech resynthesis. To generate disentangled representation, we separately extract low-bitrate representations for speech content, prosodic information, and speaker identity. This allows to synthesize speech in a controllable manner. We analyze various state-of-the-art, self-supervised representation learning methods and shed light on the advantages of each method while considering reconstruction quality and disentanglement properties. Specifically, we evaluate the F0 reconstruction, speaker identification performance (for both resynthesis and voice conversion), recordings' intelligibility, and overall quality using subjective human evaluation. Lastly, we demonstrate how these representations can be used for an ultra-lightweight speech codec. Using the obtained representations, we can get to a rate of 365 bits per second while providing better speech quality than the baseline methods. Audio samples can be found under the following link: speechbot.github.io/resynthesis.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

SoundStream: An End-to-End Neural Audio Codec (2021, methodology): “A very low-bitrate codec was proposed in [39] by decoding speech representations obtained via self-supervised learning.”
High Fidelity Neural Audio Compression (2022, methodology): “Similar methods were applied to speech resynthesis (Polyak et al., 2021), speech emotion conversion (Kreuk et al., 2021), spoken dialog system (Nguyen et al., 2022), and speech-to-speech translation (Lee et al., 2021a;b; Popuri et al., 2022).”
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, neutral): “Prior works [36, 37, 38] utilize disentangled representation for speech generation, such as speech content from self-supervised pre-trained models [39, 40, 41], fundamental frequency, and timbre, but speech quality is not satisfying.”
Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, background): “HuBERT units encode mostly content [Polyak et al., 2021], and the generated speech does not preserve the voice of the prompt.”
UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “The literature (Borsos et al. 2022; Polyak et al. 2021) reveals that semantic tokens primarily capture articulation information while lacking sufficient acoustic details, particularly in relation to speaker identity.”
Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, background): “Furthermore, recent research is focusing on leveraging latent features extracted by pretrained deep learning models (Polyak et al., 2021; Siuzdak et al., 2022).”
VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature (2022, methodology): “It has been found in the literature [30, 29] that waveform reconstruction from VQ acoustic feature needs additional prosody feature.”
A Survey on Neural Speech Synthesis (2021, methodology): “In this subsection, we review techniques on disentangling [218, 117, 275], controlling [353, 180, 240, 13, 267, 343, 192], and transferring [150, 131, 393, 6] variation information, as shown in Table 15.”
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, methodology): “Self-supervised discrete targets can disentangle linguistic content from speaker identity and prosodic information in speech [10].”
Generative Spoken Dialogue Language Modeling (2022, methodology): “For the waveform generation, we used the discrete unit-based HiFi-GAN vocoder from Polyak et al. (2021) trained on a small subset of high quality single-channel voice segments of our conversation dataset, using discrete units obtained from the Hu-BERT model and 1-hot speaker information from the…”
Text-Free Prosody-Aware Generative Spoken Language Modeling (2021, methodology): “Unfortunately, because those discovered units encode mostly phonetic information (Polyak et al., 2021), it suffers from the same prosodic information loss issue as text-based LMs.”
Direct Speech-to-Speech Translation With Discrete Units (2021, methodology): “We adopt the modified version of the HiFi-GAN neural vocoder [25] proposed in [11] for unit-to-waveform conversion.”
AudioGen: Textually Guided Audio Generation (2022, background): “…et al., 2019; Baevski et al., 2020; Wang et al., 2021), phoneme segmentation (Kreuk et al., 2020), and audio compression (Zeghidour et al., 2021; Polyak et al., 2021). van den Oord et al. (2018) and Schneider et al. (2019) suggested training a convolutional neural network to distinguish true…”
Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, methodology): “Thus, we train semantic unit-based vocoders [31] with representations clustered from (the 10th, 11th, and 12th) of the 24 HuBERT layers and report the CER of synthesized speech.”
Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations (2021, methodology): “Most recently, [35] used the discrete disentangled self-supervised representations to re-synthesize them into a waveform.”