PolyVoice: Language Models for Speech to Speech Translation

By Qianqian Dong, Zhiying Huang, Chen Xu, Yunlong Zhao, Tom Ko, Qiao Tian, Tangqui Li, Fengpeng Yue, Ye Bai, Xi Chen and 5 more. 2023

35 citations recorded by the source corpus.

We propose PolyVoice, a language model-based framework for speech-to-speech translation (S2ST) system. Our framework consists of two language models: a translation language model and a speech synthesis language model. We use discretized speech units, which are generated in a fully unsupervised way, and thus our framework can be used for unwritten languages. For the speech synthesis part, we adopt the existing VALL-E X approach and build a unit-based audio language model. This grants our framework the ability to preserve the voice characteristics and the speaking style of the original speech. We examine our system on Chinese $\rightarrow$ English and English $\rightarrow$ Spanish pairs. Experimental results show that our system can generate speech with high translation quality and audio quality. Speech samples are available at https://speechtranslation.github.io/polyvoice.

Subfields: Artificial Intelligence, Machine Translation, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Xu Tan
3. Sheng Zhao
4. Shujie Liu
5. Tao Qin
6. Yanqing Liu
7. Furu Wei
8. Tie-Yan Liu

In Machine Translation

1. Jinyu Li
2. Shujie Liu
3. Furu Wei

In Speech & Audio

1. Xu Tan
2. Jinyu Li
3. Sheng Zhao
4. Yanqing Liu
5. Shujie Liu
6. Shinji Watanabe
7. Furu Wei
8. Sanyuan Chen

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “PolyVoice [70] leveraged VALL-E to create a codec-based language model for speech-to-speech translation.”
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “For instance, PolyVoice [Dong et al., 2023] adopted VALL-E and built a codec-based language model for speech-to-speech translation.”
• VoxtLM: Unified Decoder-Only Models for Consolidating Speech Recognition, Synthesis and Speech, Text Continuation Tasks (2023, background): “They also require additional losses like an alignment loss to encourage cross-modal transfer between speech [18–20], which are similar to our approach.”
• SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models (2023, background): “…the complex information within acoustic tokens. iii) Hierarchical speech language models comprise semantic token language models and acoustic token language models, which capture content information and acoustic details respectively (Borsos et al., 2022; Rubenstein et al., 2023; Dong et al., 2023).”