Non-Parallel Training in Voice Conversion Using an Adaptive Restricted Boltzmann Machine

By Toru Nakashika, Tetsuya Takiguchi, Yasuhiro Minami. 2016

64 citations recorded by the source corpus.

In this paper, we present a voice conversion (VC) method that does not use any parallel data while training the model. VC is a technique where only speaker-specific information in source speech is converted while keeping the phonological information unchanged. Most of the existing VC methods rely on parallel data-pairs of speech data from the source and target speakers uttering the same sentences. However, the use of parallel data in training causes several problems: 1) the data used for the training are limited to the predefined sentences, 2) the trained model is only applied to the speaker pair used in the training, and 3) mismatches in alignment may occur. Although it is, thus, fairly preferable in VC not to use parallel data, a nonparallel approach is considered difficult to learn. In our approach, we achieve nonparallel training based on a speaker adaptation technique and capturing latent phonological information. This approach assumes that speech signals are produced from a restricted Boltzmann machine-based probabilistic model, where phonological information and speaker-related information are defined explicitly. Speaker-independent and speaker-dependent parameters are simultaneously trained under speaker adaptive training. In the conversion stage, a given speech signal is decomposed into phonological and speaker-related information, the speaker-related information is replaced with that of the desired speaker, and then voice-converted speech is obtained by mixing the two. Our experimental results showed that our approach outperformed another nonparallel approach, and produced results similar to those of the popular conventional Gaussian mixture models-based method that used parallel data in subjective and objective criteria.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron (2018, methodology): “To perform voice conversion, a model must synthesize an utterance, given only the acoustic signal of that utterance in a different speaker’s voice (Wu et al., 2013; Nakashika et al., 2016; Kinnunen et al., 2017; van den Oord et al., 2017; Chorowski et al., 2017).”
Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis (2018, background): “Style Transfer Style transfer is an active area of research that aims to synthesize a phrase in the prosodic style of a reference signal (Wu et al., 2013; Nakashika et al., 2016; Kinnunen et al., 2017).”
Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data (2017, background): “Learning to disentangle the mixed information from the surface representation is essential for a wide variety of speech applications: for example, noise robust speech recognition [41, 38, 37, 16], speaker verification [5], and voice conversion [40, 29, 24].”
Learning Latent Representations for Speech Generation and Transformation (2017, background): “Particularly, VAE addresses the intractability issue that occurs in even moderately complicated models such as Restricted Boltzmann machines (RBMs), which have been applied for voice conversion [9, 10, 11], and provides efficient approximated posterior inference of the latent factors.”
Normalization Driven Zero-Shot Multi-Speaker Speech Synthesis (2021, methodology): “…generated outputs from ZSM-SS architecture are given in 1 Apart from subjective evaluation, we have used the metrics namely Gross Pitch Error(GPE) [31], Voicing Decision Error(VDE) [31], F0 Frame Error(FFE) [32], Mel Cepstral Distortion(MCD) [33] which are used in audio signal processing to…”
High-Quality Nonparallel Voice Conversion Based on Cycle-Consistent Adversarial Network (2018, methodology): “[22] proposed a more accurate method in which an adaptive restricted Boltzmann machine uses weights composed of both common weights and speaker identity weights.”
CycleGAN-VC: Non-parallel Voice Conversion Using Cycle-Consistent Adversarial Networks (2018, methodology): “A few attempts [40]–[43] have recently been made to develop methods that are completely free from parallel data and extra modules, similarly to ours.”
An overview of voice conversion systems (2017, methodology): “During onversion, the phonetic and speaker information are separated nd the speaker information is replaced with that of the target’s Nakashika et al., 2016 ).”
Non-parallel voice conversion using i-vector PLDA: towards unifying speaker verification and transformation (2017, methodology): “[16]), it is inspired from ASV, including the use of conventional mel-frequency cepstral coefficients for vocoding.”