Disentangling Correlated Speaker and Noise for Speech Synthesis via Data Augmentation and Adversarial Factorization

By Wei-Ning Hsu, Yu Zhang, Ron J. Weiss, Yu-An Chung, Yuxuan Wang, Yonghui Wu, James R. Glass. 2019

125 citations recorded by the source corpus.

To leverage crowd-sourced data to train multi-speaker text-to-speech (TTS) models that can synthesize clean speech for all speakers, it is essential to learn disentangled representations which can independently control the speaker identity and background noise in generated signals. However, learning such representations can be challenging, due to the lack of labels describing the recording conditions of each training example, and the fact that speakers and recording conditions are often correlated, e.g. since users often make many recordings using the same equipment. This paper proposes three components to address this problem by: (1) formulating a conditional generative model with factorized latent variables, (2) using data augmentation to add noise that is not correlated with speaker identity and whose label is known during training, and (3) using adversarial factorization to improve disentanglement. Experimental results demonstrate that the proposed method can disentangle speaker and noise attributes even if they are correlated in the training data, and can be used to consistently synthesize clean speech for all speakers. Ablation studies verify the importance of each proposed component.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

A Survey on Neural Speech Synthesis (2021, methodology): “Since TTS is a typical sequence generation task and can be modeled through typical generative models, we can categorize in terms of different generative models: normal sequence generation model, flow, GAN, VAE, and diffusion model.”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, methodology): “by a synthesis model benefits from training data recorded by a single speaker in a clean recording environment [53, 110, 29, 52].”
Feedback Adversarial Learning: Spatial Feedback for Improving Generative Adversarial Networks (2019, background): “GANs have continued to show impressive results in various applications such as image generation [46, 15, 63, 8], text to image synthesis [47, 26], future frame prediction [40, 55], image editing [67], novel view synthesis [52, 43], domain adaptation [7, 51], 3D modeling [59, 27], video generation [6”
Noise Robust TTS for Low Resource Speakers using Pre-trained Model and Speech Enhancement (2020, methodology): “To leverage low-quality crowd-sourced data to train multispeaker TTS models that can synthesize clean speech for all speakers, based on Tacotron2, [11] introduced conditional generative reference encoders and adversarial training to learn disentangled representations to independently control the spe”
Fully-Hierarchical Fine-Grained Prosody Modeling For Interpretable Speech Synthesis (2020, background): “Although most of these works use latent representations at utterance level which captures the salient features of the utterance [6, 10–12], fine-grained prosody that are aligned with the phone sequence can be captured using techniques recently proposed in [13].”
Disentangled Speaker and Language Representations Using Mutual Information Minimization and Domain Adaptation for Cross-Lingual TTS (2021, background): “Recent end-to-end multilingual and multi-speaker text-to-speech (TTS) synthesis systems typically rely on distributed speaker and language representations to model different speakers and languages [1, 2, 3, 4, 5, 6].”
Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models (2019, methodology): “For instance, instead of utilizing only knowledge from unlabeled text data, we can also make use of large amounts of unlabeled speech corpora [26] to assist the decoder in learning the acoustic representations and alignments.”
Effective and Differentiated Use of Control Information for Multi-speaker Speech Synthesis (2021, background): “[21] aimed to disentangle speaker embedding and noise by data augmentation and a conditional generative model.”
Whispered and Lombard Neural Speech Synthesis (2021, background): “[27, 28, 29] have proposed extending the Tacotron to a multi-speaker model by conditioning the synthesis on a speaker representation in addition to the phoneme sequence.”
Can Speaker Augmentation Improve Multi-Speaker End-to-End TTS? (2020, methodology): “This is different from previous work, which uses low-quality recordings for speaker adaptation [10, 11, 12] and multi-speaker modeling (e.”
Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “[76] for modeling global features of speech such as different prosodic patterns of different speakers, and the other similar to Sun et al.”
Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding (2020, unknown): “…studies proposed several specialized embeddings, each with a different set of respon-sibilities to represent diverse speech attributes (e.g., speaking style prosody, and noise) [15, 16, 17, 18, 19, 20] or switched over to a variable-length embedding to maintain temporal information [21, 22, 23].”
Neural Text-to-Speech Adaptation from Low Quality Public Recordings (2019, background): “Adding an additional encoder to decrease the noise [15] makes the network difﬁcult to optimize.”
Denoispeech: Denoising Text to Speech with Frame-Level Noise Modeling (2020, methodology): “…noisy corpus or the ground-truth noisy recordings of real-world noisy corpus; 3) Augment-Adversarial , apply data augmentation and adversarial factorization to FastSpeech 2 as used in [7] ; 4) Enhancement-Based , FastSpeech 2 trained on enhanced speech denoised with an enhancement model [5].”
The Multi-Speaker Multi-Style Voice Cloning Challenge 2021 (2021, unknown): “Besides, in real-world voice cloning, low quality recording or noisy speech is another issue to address [21, 22].”
STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speech (2021, methodology): “In the TTS domain, this has been tackled by applying Domain Adversarial Training (DAT) [24, 25].”
Data Efficient Voice Cloning from Noisy Samples with Domain Adversarial Training (2020, background): “And in [15], through speaker and noise attributes disentangling during training, the model is able to control different aspects of the synthesized speech with different reference audios.”
Pretraining Strategies, Waveform Model Choice, and Acoustic Configurations for Multi-Speaker End-to-End Speech Synthesis (2020, methodology): “Variational autoencoder based clean speech and noise factorization has also been proposed as a way to make use of both noisy and clean data together, with [12] using artificially-corrupted speech and [13] using real noisy speech, to create speaker-adapted TTS voices with clean output conditions.”