Voice Impersonation Using Generative Adversarial Networks

By Yang Gao, Rita Singh, Bhiksha Raj. 2018

108 citations recorded by the source corpus.

Voice impersonation is not the same as voice transformation, although the latter is an essential element of it. In voice impersonation, the resultant voice must convincingly convey the impression of having been naturally produced by the target speaker, mimicking not only the pitch and other perceivable signal qualities, but also the style of the target speaker. In this paper, we propose a novel neural-network based speech quality- and style-mimicry framework for the synthesis of impersonated voices. The framework is built upon a fast and accurate generative adversarial network model. Given spectrographic representations of source and target speakers' voices, the model learns to mimic the target speaker's voice quality and style, regardless of the linguistic content of either's voice, generating a synthetic spectrogram from which the time-domain signal is reconstructed using the Griffin-Lim method. In effect, this model reframes the well-known problem of style-transfer for images as the problem of style-transfer for speech signals, while intrinsically addressing the problem of durational variability of speech sounds. Experiments demonstrate that the model can generate extremely convincing samples of impersonated speech. It is even able to impersonate voices across different genders effectively. Results are qualitatively evaluated using standard procedures for evaluating synthesized voices.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

Generative Adversarial Networks (2021, neutral): “Inconceivably, we can even modify the linguistic content by using the VoiceGAN model [13].”
The Zero Resource Speech Challenge 2019: TTS without T (2019, neutral): “The ZR19 task is also similar to the voice conversion problem [16], in which the audio of a given source speaker is converted to a target speaker, without annotation (using VAEs [17], disentangling autoencoders [18], or GANs [19]).”
Zero-Shot Voice Style Transfer with Only Autoencoder Loss (2019, methodology): “GAN alone is also applied to voice conversion (Gao et al., 2018).”
Virtual Phone Discovery for Speech Synthesis Without Text (2019, methodology): “Recently, techniques have been developed without the use of parallel data for voice conversion [16, 17, 18, 19].”
Any-to-Many Voice Conversion With Location-Relative Sequence-to-Sequence Modeling (2020, background): “generative adversarial networks [17]–[20] and their combinations [21]–[23].”
Unsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversion (2019, background): “…also attempted to encode speech content into various representations for voice conversion, including the use of disentangled autoencoders [11], VAEs [12] or GANs [13]. continuous vectors are the most common approach, however when it comes to encoding speech content, they may not be the best choice.”
Multi-target Voice Conversion without Parallel Data by Adversarially Learning Disentangled Audio Representations (2018, methodology): “Cycle-consistent adversarial network (Cycle-GAN) was also used to learn the mapping from the source speaker to the target speaker in an unsupervised way [25, 26].”
Unsupervised Speech Decomposition via Triple Information Bottleneck (2020, methodology): “Inspired by image style transfer frameworks, Gao et al. (2018) and Kameoka et al. (2018b) adapted CycleGan (Kaneko & Kameoka, 2017) and StarGan (Choi et al., 2018) respectively for voice conversion.”