Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling
By Zi-Hua Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li and 3 more. 2023
275 citations recorded by the source corpus.
We propose a cross-lingual neural codec language model, VALL-E X, for cross-lingual speech synthesis. Specifically, we extend VALL-E and train a multi-lingual conditional codec language model to predict the acoustic token sequences of the target language speech by using both the source language speech and the target language text as prompts. VALL-E X inherits strong in-context learning capabilities and can be applied for zero-shot cross-lingual text-to-speech synthesis and zero-shot speech-to-speech translation tasks. Experimental results show that it can generate high-quality speech in the target language via just one speech utterance in the source language as a prompt while preserving the unseen speaker's voice, emotion, and acoustic environment. Moreover, VALL-E X effectively alleviates the foreign accent problems, which can be controlled by a language ID. Audio samples are available at \url{https://aka.ms/vallex}.
Subfields: Artificial Intelligence, Machine Translation, Speech & Audio
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Machine Translation
In Speech & Audio
Citations in the corpus
- Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “VALL-EX[42]extendedVALL-Etocross-lingualzero-shotTTStasks.”
- BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, methodology): “We proposed a new way to measure textual understanding of TTS models, and showed that LTTS models built with 10K hours of data and 400 million parameters start to exhibit an advanced grasp of text that enables contextually appropriate prosody.”
- ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, background): “VALL-E-X (Zhang et al., 2023) extends VALL-E by utilizing source language speech and target language text as prompts when predicting the acoustic marker sequence of the target language speech.”
- UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, background): “…tokenized as sequences Zeghidour et al. (2021) that can be further processed by LMs. LM-based audio generative models then show superior capability in generalization towards unseen speakers Wang et al. (2023a), low resources (Kharitonov et al., 2023) and multilingual (Zhang et al., 2023) scenarios.”
- SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, neutral): “In contrast, recent approaches have embraced more generic formulations, such as masked speech prediction [15] or neural codec language modeling [16], [17], [18], [19].”
- Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “4) Content has monotonic alignment with speech while the autoregressive language model cannot guarantee that, which can lead to repeating or missing word issues [59, 58, 67].”
- PolyVoice: Language Models for Speech to Speech Translation (2023, methodology): “For the speech synthesis part, we adopt the VALL-E X approach (Zhang et al., 2023) for the voice clone ability.”
- VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation (2023, neutral): “Moreover, researchers also apply the decoder-only model to text-to-speech synthesis tasks, namely VALL-E (Wang et al., 2023), and VALL-E X (Zhang et al., 2023b).”
- Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024, neutral): “Zhang et al. (2023) train a multilingual conditional codec language model to predict the acoustic token sequences of speech in different languages.”
- Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, background): “Given that previous AR models [39, 46] have addressed intelligibility challenges in generated samples through large-scale training, we have incorporated the autoregressive approach into our model design.”
- Discrete Audio Representation as an Alternative to Mel-Spectrograms for Speaker and Speech Recognition (2023, methodology): “One class of foundational audio model efforts like VALL-E [8], VALL-E X [9], VI-OLA [10], MusicLM [11] use compression based tokens where as models such as AudioLM [12], AudioPaLM [13] employ semantic tokens.”
- F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (2024, background): “With a few seconds of audio prompt, current TTS models are able to synthesize speech for any given text and mimic the speaker of audio prompt [9, 10].”
- Scaling Speech Technology to 1, 000+ Languages (2023, background): “Speech synthesis has been undergoing a transition from controlled settings to the generation of more diverse speech, with multilinguality being a crucial aspect [Casanova et al., 2022, Zhang et al., 2023b].”
- Mega-TTS 2: Zero-Shot Text-to-Speech with Arbitrary Length Speech Prompts (2023, background): “VALL-E X(Zhang et al., 2023) extends VALL-E by introducing language ID to support cross-lingual TTS and speech-to-speech translation.”
- The rise and potential of large language model based agents: a survey (2025, unknown): “Beyond texts, researchers also explore the potential ICL capabilities in different multimodal tasks [189–194], making it possible for agents to be applied to large-scale real-world tasks.”
- SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities (2023, methodology): “…spoken language models involves encoding the speech signal into a discrete representation (Baevski et al., 2020; Hsu et al., 2021; Zhang et al., 2023a) and modeling it with language models (Lakhotia et al., 2021; Borsos et al., 2022; Zhang et al., 2023d; Wang et al., 2023; Zhang et al., 2023c).”