UWSpeech: Speech to Speech Translation for Unwritten Languages

By Chen Zhang, Xu Tan, Yi Ren, Tao Qin, Ke-jun Zhang, Tie-Yan Liu. 2020

71 citations recorded by the source corpus.

Existing speech to speech translation systems heavily rely on the text of target language: they usually translate source language either to target text and then synthesize target speech from text, or directly to target speech with target text for auxiliary training. However, those methods cannot be applied to unwritten target languages, which have no written text or phoneme available. In this paper, we develop a translation system for unwritten languages, named as UWSpeech, which converts target unwritten speech into discrete tokens with a converter, and then translates source-language speech into target discrete tokens with a translator, and finally synthesizes target speech from target discrete tokens with an inverter. We propose a method called XL-VAE, which enhances vector quantized variational autoencoder (VQ-VAE) with cross-lingual (XL) speech recognition, to train the converter and inverter of UWSpeech jointly. Experiments on Fisher Spanish-English conversation translation dataset show that UWSpeech outperforms direct translation and VQ-VAE baseline by about 16 and 10 BLEU points respectively, which demonstrate the advantages and potentials of UWSpeech.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Machine Translation, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Wei-Ning Hsu
5. Yossi Adi
6. Juan Pino
7. Zhou Zhao
8. Changhan Wang

In Computer Vision and Pattern Recognition

1. Tao Qin
2. Tie-Yan Liu
3. Luc Van Gool
4. Dacheng Tao
5. Xu Tan
6. Xiaogang Wang
7. Jitendra Malik
8. Ming-Hsuan Yang

In Machine Translation

1. Ye Jia
2. Juan Pino
3. Changhan Wang
4. Yun Tang
5. Yossi Adi
6. Anne Wu
7. Xutai Ma
8. Ann Lee

In Speech & Audio

1. Xu Tan
2. Wei-Ning Hsu
3. Yossi Adi
4. Tao Qin
5. Yi Ren
6. Tie-Yan Liu
7. Zhou Zhao
8. Ye Jia

Citations in the corpus

• VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, background): “Finally, there has been little research (Jia et al., 2019; Tjandra et al., 2019; Zhang et al., 2020a) into speech output due to the lack of open data.”
• AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “There are other cascaded S2ST systems that utilize learned discrete speech representations as an intermediate representation [Tjandra et al., 2019, Zhang et al., 2021, Lee et al., 2022, Ma et al., 2021, Lee et al., 2021].”
• PolyVoice: Language Models for Speech to Speech Translation (2023, methodology): “And we utilize a subset of 10,000 hours of GigaSpeech (Chen et al., 2021), an English ASR dataset collected from audiobooks, podcasts, and YouTube.”
• A Survey on Neural Speech Synthesis (2021, background): “Since human languages share similar vocal organs, pronunciations [383] and semantic structures [335], pre-training the TTS models on rich-resource languages can help the mapping between text and speech in low-resource languages [42, 390, 12, 60, 98, 336, 435, 241, 428].”
• Textless Speech-to-Speech Translation on Real Data (2021, neutral): “…a speech retrieval task (Zanon Boito et al., 2020) or human evaluation of the data quality (Duquenne et al., 2021), since existing direct S2ST systems are mostly trained with synthetic target speech (Jia et al., 2019; Tjan-dra et al., 2019; Zhang et al., 2020; Lee et al., 2021; Jia et al., 2021).”
• Direct Speech-to-Speech Translation With Discrete Units (2021, methodology): “In addition, direct S2ST is a natural approach for supporting translation for languages that do not have a writing system [3, 4].”
• CVSS Corpus and Massively Multilingual Speech-to-Speech Translation (2022, neutral): “, 2021a) and cascade S2ST based on discrete speech representation (Tjandra et al., 2019; Zhang et al., 2021; Lee et al., 2021a; Ma et al., 2021; Lee et al., 2021b).”
• TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation (2022, background): “UWSpeech [5] builds the vector-quantized variational auto-encoder (VQ-VAE) model and discards transcript in the target language, while paired speech and phoneme corpus of written language is required.”
• Textless Direct Speech-to-Speech Translation with Discrete Speech Representation (2022, methodology): “Unlike previous works using discrete representations [14,17], which require multiple separately trained models (e.g: translation model and vocoder) to be cascaded, our proposed model can be trained end-to-end.”
• Translatotron 2: High-quality direct speech-to-speech translation with voice preservation (2021, methodology): “, 2021) and cascade S2ST based on discrete speech representation (Tjandra et al., 2019; Zhang et al., 2021; Lee et al., 2022; 2021a; Ma et al., 2021).”
• DUB: Discrete Unit Back-translation for Speech Translation (2023, background): “…(Sicher-man and Adi, 2023; Lakhotia et al., 2021), leading to significant improvements in the speech tasks, such as automatic speech recognition (Meng et al., 2022), text-to-speech (Dunbar et al., 2019), and speech-to-speech translation (Zhang et al., 2021; Lee et al., 2022; Inaguma et al., 2022).”
• Variable-rate discrete representation learning (2021, methodology): “Besides the work by Hayashi & Watanabe (2020) mentioned earlier, this approach has been used for speech-to-speech translation (Zhang et al., 2020), video representation learning (Sun et al.”