Unsupervised Speech Decomposition via Triple Information Bottleneck

By Kaizhi Qian, Yang Zhang, Shiyu Chang, David D. Cox, Mark Hasegawa-Johnson. 2020

212 citations recorded by the source corpus.

Speech information can be roughly decomposed into four components: language content, timbre, pitch, and rhythm. Obtaining disentangled representations of these components is useful in many speech analysis and generation applications. Recently, state-of-the-art voice conversion systems have led to speech representations that can disentangle speaker-dependent and independent information. However, these systems can only disentangle timbre, while information about pitch, rhythm and content is still mixed together. Further disentangling the remaining speech components is an under-determined problem in the absence of explicit annotations for each component, which are difficult and expensive to obtain. In this paper, we propose SpeechSplit, which can blindly decompose speech into its four components by introducing three carefully designed information bottlenecks. SpeechSplit is among the first algorithms that can separately perform style transfer on timbre, pitch and rhythm without text labels. Our code is publicly available at this https URL.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, background): “…prosody, timbre, and acoustic details and reconstruct speech waveform with these disentangled representations, leveraging information bottleneck [16, 17], various supervised losses, and adversarial training [18] to enhance disentanglement; 2) we propose a factorized diffusion model, which…”
A Survey on Neural Speech Synthesis (2021, methodology): “In this subsection, we review techniques on disentangling [218, 117, 275], controlling [353, 180, 240, 13, 267, 343, 192], and transferring [150, 131, 393, 6] variation information, as shown in Table 15.”
Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, background): “Voice synthesis [40, 33, 32] aims to generate human-like voices, which attracts broad interest in the machine learning community.”
Review of end-to-end speech synthesis technology based on deep learning (2021, background): “[166] proposed SPEECHFLOW, which is a speech style conversion model that can disentangle the rhythm, pitch, content, and timbre information.”
SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models (2023, methodology): “These approaches adopt a parallel disentanglement strategy, where the speech is fed into parallel content and speaker encoders to obtain different representations (Qian et al., 2020).”
TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation (2022, methodology): “UWSpeech (Zhang et al., 2020) builds the VQ-VAE model and discards transcript in the target language, while paired speech and phoneme corpora of written language are required.”
Improved Prosody from Learned F0 Codebook Representations for VQ-VAE Speech Waveform Reconstruction (2020, methodology): “Another related extension comes from SPEECHSPLIT [13] wherein three different auto-encoders were used to disentangle the speech signal into components, however their aim was to extract continuous latents.”
STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speech (2021, methodology): “In the unsupervised condition [19, 20], encoders need a tight bottleneck both channel-wise and frame-wise to deliver the necessary information from the source audio.”
Textless Speech Emotion Conversion using Decomposed and Discrete Representations (2021, background): “…such as temporal smoothness (Ebbers et al., 2017; Glarner et al., 2018; Khurana et al., 2020), discreteness (Ondel et al., 2016; van den Oord et al., 2017), and presence of hierarchy (Hsu et al., 2017), information bottlenecks for speech representation decomposition (Qian et al., 2020, 2021).”
LM-VC: Zero-Shot Voice Conversion via Speech Generation Based on Language Models (2023, methodology): “Some studies [2], [5] also use signal perturbation techniques to alter speech utterances to make it speaker irrelevant before content extraction.”
Chunked Autoregressive GAN for Conditional Waveform Synthesis (2021, background): “…audio waveform generation has seen remarkable improvements in ﬁdelity and computational cost in recent years, beneﬁting applications such as text-to-speech (Kim et al., 2021), voice editing (Qian et al., 2020; Morrison et al., 2020), and controllable music generation (Dhari-wal et al., 2020).”
GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis (2022, background): “It (Weston et al., 2021; Qian et al., 2020) has been demonstrated that except the input text, there exist many sources of variation in speech, including speaker identity, emotion, and prosody.”
Disentangling speech from surroundings in a neural audio codec (2022, methodology): “To avoid issues of adversarial training, some works rely on constraining the information capacity by a careful choice of embedding dimensions that ensures each channel contains only one factor of variation, such as AutoVC [17] and SpeechSplit [18].”
How Far Are We from Robust Voice Conversion: A Survey (2020, background): “Zeroshot approaches [8, 9, 10, 11, 12] are then considered, where the models can perform VC among any speakers without fine-tuning.”
Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversion (2022, methodology): “Speech representation encoders: Inspired by SpeechSplit [25], the rhythm encoderEr , pitch encoderEp, content encoder Ec have the same structure, which consists of a stack of 5 × 1 convolutional layers and a stack of bidirectional long short-term memory (BiLSTM) layers.”