LM-VC: Zero-Shot Voice Conversion via Speech Generation Based on Language Models
By Zhichao Wang, Yuan-Jui Chen, Linfu Xie, Qiao Tian, Yuping Wang. 2023
47 citations recorded by the source corpus.
Language model (LM) based audio generation frameworks, e.g., AudioLM, have recently achieved new state-of-the-art performance in zero-shot audio generation. In this paper, we explore the feasibility of LMs for zero-shot voice conversion. An intuitive approach is to follow AudioLM – Tokenizing speech into semantic and acoustic tokens respectively by HuBERT and SoundStream, and converting source semantic tokens to target acoustic tokens conditioned on acoustic tokens of the target speaker. However, such an approach encounters several issues: 1) the linguistic content contained in semantic tokens may get dispersed during multi-layer modeling while the lengthy speech input in the voice conversion task makes contextual learning even harder; 2) the semantic tokens still contain speaker-related information, which may be leaked to the target speech, lowering the target speaker similarity; 3) the generation diversity in the sampling of the LM can lead to unexpected outcomes during inference, leading to unnatural pronunciation and speech quality degradation. To mitigate these problems, we propose LM-VC, a two-stage language modeling approach that generates coarse acoustic tokens for recovering the source linguistic content and target speaker's timbre, and then reconstructs the fine for acoustic details as converted speech. Specifically, to enhance content preservation and facilitates better disentanglement, a masked prefix LM with a mask prediction strategy is used for coarse acoustic modeling. This model is encouraged to recover the masked content from the surrounding context and generate target speech based on the target speaker's utterance and corrupted semantic tokens. Besides, to further alleviate the sampling error in the generation, an external LM, which employs window attention to capture the local acoustic relations, is introduced to participate in the coarse acoustic modeling through shallow fusion. Finally, a prefix LM reconstructs fine acoustic tokens from the coarse and results in the converted speech. Experiments demonstrate that LM-VC outperforms competitive systems in speech naturalness and speaker similarity.
Subfields: Artificial Intelligence, Physiology, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Signal Processing
Citations in the corpus
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, neutral): “…adv L adv + λ feat L feat + λ codebook L codebook + λ commit L commit + λ ph L ph + λ f0 L f0 + λ gr-ph L gr-ph + λ gr-f0 L gr-f0 + λ gr-spk L gr-spk , where λ rec , λ adv , λ feat , λ codebook , λ commit , λ f0 , λ ph , λ gr-f0 , λ gr-ph and λ gr-spk are coefficients for balancing each loss terms.”
- UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, background): “…are represented as Task Conditions Audio Target Text-to-Speech (TTS) (Wang et al., 2023a) phoneme, speaker prompt speech Voice Conversion (VC) ♣ (Wang et al., 2023e) semantic token, speaker prompt speech Speech Enhancement (SE) ♣ (Wang et al., 2023b) noisy speech speech Target Speech Extraction…”
- Seed-TTS: A Family of High-Quality Versatile Speech Generation Models (2024, methodology): “Most prior approaches achieve attribute disentanglement through feature engineering [Chen et al., 2023, Wang et al., 2023a, Liu et al., 2021, Anastassiou et al., 2024, Lee et al., 2023, Choi et al., 2024], specific loss functions [Ju et al., 2024, Jia et al., 2022], or precise network architecture…”