Multi-target Voice Conversion without Parallel Data by Adversarially Learning Disentangled Audio Representations

By Ju-Chieh Chou, Cheng-chieh Yeh, Hung-yi Lee, Lin-Shan Lee. 2018

137 citations recorded by the source corpus.

Recently, cycle-consistent adversarial network (Cycle-GAN) has been successfully applied to voice conversion to a different speaker without parallel data, although in those approaches an individual model is needed for each target speaker. In this paper, we propose an adversarial learning framework for voice conversion, with which a single model can be trained to convert the voice to many different speakers, all without parallel data, by separating the speaker characteristics from the linguistic content in speech signals. An autoencoder is first trained to extract speaker-independent latent representations and speaker embedding separately using another auxiliary speaker classifier to regularize the latent representation. The decoder then takes the speaker-independent latent representation and the target speaker embedding as the input to generate the voice of the target speaker with the linguistic content of the source utterance. The quality of decoder output is further improved by patching with the residual signal produced by another pair of generator and discriminator. A target speaker set size of 20 was tested in the preliminary experiments, and very good voice quality was obtained. Conventional voice conversion metrics are reported. We also show that the speaker information has been properly reduced from the latent representations.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks (2019, background): “This is related to voice conversion [21, 22], where speech from one speaker needs to be synthesised in another speaker’s voice.”
The Zero Resource Speech Challenge 2019: TTS without T (2019, neutral): “The ZR19 task is also similar to the voice conversion problem [16], in which the audio of a given source speaker is converted to a target speaker, without annotation (using VAEs [17], disentangling autoencoders [18], or GANs [19]).”
Unsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversion (2019, methodology): “We compare the proposed MBV encodings with one-hot encodings, continuous encodings, and continuous encodings with additional loss [11], all of which under the same autoencoder training setting as described in Section 3.”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, methodology): “Recent GAN-based approaches, such as CycleGAN [253–256], VAW-GAN [257], and StarGAN [258], attempt to achieve high-quality transformed speech using non-parallel training data.”
StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion (2021, methodology): “For a fair comparison, we train both the baseline model and our framework on the same 20 selected speakers reported in [18] from VCTK dataset [19].”
How Far Are We from Robust Voice Conversion: A Survey (2020, methodology): “DGAN-VC [6] adopts a two-stage training framework for many-to-many voice conversion.”
Unsupervised Speech Decomposition via Triple Information Bottleneck (2020, background): “Chou et al. (2018) introduced a classifer for the latent code, and discourages the latent code to be correctly classifed as the target speaker.”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, methodology): “Voice conversion has recently seen progress using neural approaches [45, 49, 50, 46, 27, 12, 14, 69, 56, 87], but the most relevant work to our own is the ZeroSpeech 2019 challenge [24, 94, 10], which explicitly addresses the unsupervised learning of discrete speech units which can be used as the ba”
Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge (2020, background): “While similar to voice conversion [7, 8], an explicit goal of ZeroSpeech 2019 is to learn low-bitrate representations that perform well on phone discrimination tests.”
Towards Robust Neural Vocoding for Speech Generation: A Survey (2019, methodology): “Most speech generation models, such as text-to-speech [1, 2, 3, 4] and voice conversion [5, 6, 7, 8], do not output waveform directly.”
WG-WaveNet: Real-Time High-Fidelity Speech Synthesis without GPU (2020, background): “Recently, neural network-based models have achieved state-ofthe-art performance in speech tasks such as text-to-speech and voice conversion [1, 2, 3, 4].”
Zero-Shot Voice Style Transfer with Only Autoencoder Loss (2019, methodology): “Chou et al. (2018) introduce a classiﬁer on the code and a GAN on the output.”
Upsampling Artifacts in Neural Audio Synthesis (2020, methodology): “Three main types of neural upsamplers exist: transposed convolutions [1, 3, 9], interpolation upsamplers [2, 11], and subpixel convolutions [10,12,13].”