Any-to-Many Voice Conversion With Location-Relative Sequence-to-Sequence Modeling

By Songxiang Liu, Yuewen Cao, Disong Wang, Xixin Wu, Xunying Liu, Helen Mei-Ling Meng. 2020

125 citations recorded by the source corpus.

This paper proposes an any-to-many location-relative, sequence-to-sequence (seq2seq), non-parallel voice conversion approach, which utilizes text supervision during training. In this approach, we combine a bottle-neck feature extractor (BNE) with a seq2seq synthesis module. During the training stage, an encoder-decoder-based hybrid connectionist-temporal-classification-attention (CTC-attention) phoneme recognizer is trained, whose encoder has a bottle-neck layer. A BNE is obtained from the phoneme recognizer and is utilized to extract speaker-independent, dense and rich spoken content representations from spectral features. Then a multi-speaker location-relative attention based seq2seq synthesis model is trained to reconstruct spectral features from the bottle-neck features, conditioning on speaker representations for speaker identity control in the generated speech. To mitigate the difficulties of using seq2seq models to align long sequences, we down-sample the input spectral feature along the temporal dimension and equip the synthesis model with a discretized mixture of logistic (MoL) attention mechanism. Since the phoneme recognizer is trained with large speech recognition data corpus, the proposed approach can conduct any-to-many voice conversion. Objective and subjective evaluations show that the proposed any-to-many approach has superior voice conversion performance in terms of both naturalness and speaker similarity. Ablation studies are conducted to confirm the effectiveness of feature selection and model design strategies in the proposed approach. The proposed VC approach can readily be extended to support any-to-any VC (also known as one/few-shot VC), and achieve high performance according to objective and subjective evaluations.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dong Yu
2. Chao Weng
3. Rongjie Huang
4. Zhou Zhao
5. Dongchao Yang
6. Jinyu Li
7. Songxiang Liu
8. Xuankai Chang

In Signal Processing

1. Rongjie Huang
2. Dong Yu
3. Chao Weng
4. Dongchao Yang
5. Zhou Zhao
6. Jinyu Li
7. Xuankai Chang
8. Songxiang Liu

Citations in the corpus

• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, methodology): “…auditory perception (Shadle & Damper, 2001) and machine learning (Wang et al., 2016) research etc; (4) they use different generative model, such as diffusion model (Huang et al., 2023a; Shen et al., 2023; Wang et al., 2023b), flow (Le et al., 2023), Seq2Seq (Ren et al., 2020; Liu et al., 2021).”
• Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024, neutral): “We compare the generated audio samples with other systems, including 1) GT, the ground-truth audio; 2) YourTTS (Casanova et al., 2022), Gen-erSpeech (Huang et al., 2022b), VALL-E (Wang et al., 2023a) for English zero-shot TTS; 3) YourTTS (Casanova et al., 2022) for zero-shot multilingual TTS; 4) NAN”
• Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, methodology): “We compare the generated audio samples with other systems, including 1) GT, the ground-truth audio; 2) NANSY [7]: the unified framework of synthesizing and manipulating voice signals from analysis features; 3) PPG-VC [26]: the voice conversion model based on phonetic posterior-gram.”
• HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “The speech synthesizer synthesizes 16-kHz speech with a target voice style from the target voice prompt.”
• Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme (2021, methodology): “The proposed model Diff-LibriTTS-ML-30 and the baseline BNE-PPG-VC show the same performance on the VCTK test set in terms of speech naturalness the latter being slightly better in terms of speaker similarity which can perhaps be explained by the fact that BNE-PPG-VC was trained on the union of VCTK”