Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling

By Zi-Hua Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li and 3 more. 2023

275 citations recorded by the source corpus.

We propose a cross-lingual neural codec language model, VALL-E X, for cross-lingual speech synthesis. Specifically, we extend VALL-E and train a multi-lingual conditional codec language model to predict the acoustic token sequences of the target language speech by using both the source language speech and the target language text as prompts. VALL-E X inherits strong in-context learning capabilities and can be applied for zero-shot cross-lingual text-to-speech synthesis and zero-shot speech-to-speech translation tasks. Experimental results show that it can generate high-quality speech in the target language via just one speech utterance in the source language as a prompt while preserving the unseen speaker's voice, emotion, and acoustic environment. Moreover, VALL-E X effectively alleviates the foreign accent problems, which can be controlled by a language ID. Audio samples are available at \url{https://aka.ms/vallex}.

Subfields: Artificial Intelligence, Machine Translation, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Dong Yu
  2. Wei-Ning Hsu
  3. Jinyu Li
  4. James R. Glass
  5. Yoshua Bengio
  6. Tie-Yan Liu
  7. Mohit Bansal
  8. Tao Qin

In Machine Translation

  1. Yossi Adi
  2. Dong Yu
  3. Michael Auli
  4. Alexis Conneau
  5. Wei-Ning Hsu
  6. Mingxuan Wang
  7. Alexei Baevski

In Speech & Audio

  1. Wei-Ning Hsu
  2. Jinyu Li
  3. Rongjie Huang
  4. Yossi Adi
  5. Zhou Zhao
  6. Yi Ren
  7. Alexis Conneau
  8. Dong Yu

Citations in the corpus