Speech-to-Speech Translation Between Untranscribed Unknown Languages

By Andros Tjandra, Sakriani Watiasri Sakti, Satoshi Nakamura. 2019

56 citations recorded by the source corpus.

In this paper, we explore a method for training speech-to-speech translation tasks without any transcription or linguistic supervision. Our proposed method consists of two steps: First, we train and generate discrete representation with unsupervised term discovery with a discrete quantized autoencoder. Second, we train a sequence-to-sequence model that directly maps the source language speech to the target languages discrete representation. Our proposed method can directly generate target speech without any auxiliary or pre-training steps with a source or target transcription. To the best of our knowledge, this is the first work that performed pure speech-to-speech translation between untranscribed unknown languages.

Subfields: Machine Translation, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Machine Translation

1. Ye Jia
2. Juan Pino
3. Changhan Wang
4. Yun Tang
5. Yossi Adi
6. Anne Wu
7. Xutai Ma
8. Ann Lee

In Speech & Audio

1. Wei-Ning Hsu
2. Yossi Adi
3. Xu Tan
4. James R. Glass
5. Ye Jia
6. Tao Qin
7. Juan Pino
8. Changhan Wang

Citations in the corpus

• VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, background): “Finally, there has been little research (Jia et al., 2019; Tjandra et al., 2019; Zhang et al., 2020a) into speech output due to the lack of open data.”
• AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “There are other cascaded S2ST systems that utilize learned discrete speech representations as an intermediate representation [Tjandra et al., 2019, Zhang et al., 2021, Lee et al., 2022, Ma et al., 2021, Lee et al., 2021].”
• PolyVoice: Language Models for Speech to Speech Translation (2023, methodology): “, 2021) extracted from the speech are used to replace the target text, and then can be synthesized into the speech (Tjandra et al., 2019; Zhang et al., 2021; Lee et al., 2022).”
• Textless Speech-to-Speech Translation on Real Data (2021, neutral): “The systems are trained either without any text data (Tjandra et al., 2019) or with text data from other languages (Zhang et al.”
• Direct Speech-to-Speech Translation With Discrete Units (2021, methodology): “[3, 4] both investigate direct S2ST models under the unwritten language setup by transforming the target speech into discrete representations through Variational Auto-Encoder (VAE), training a sequence-to-sequence model for translation into target discrete units, and an inverter for converting the u”
• UWSpeech: Speech to Speech Translation for Unwritten Languages (2020, background): “2019) and the more advanced vector quantized VAE (VQ-VAE) (van den Oord, Vinyals et al. 2017; Dunbar et al. 2019; Tjandra et al. 2019; Chorowski et al. 2019; Liu et al. 2019; Tjandra, Sakti, and Nakamura 2019; Baevski, Schneider, and Auli 2019).”
• CVSS Corpus and Massively Multilingual Speech-to-Speech Translation (2022, neutral): “, 2021a) and cascade S2ST based on discrete speech representation (Tjandra et al., 2019; Zhang et al., 2021; Lee et al., 2021a; Ma et al., 2021; Lee et al., 2021b).”
• Textless Direct Speech-to-Speech Translation with Discrete Speech Representation (2022, background): “A few works have demonstrated the feasibility of training S2ST system without textual supervision, they significantly underperform similar approaches when textual supervision is used [14–16].”
• Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, background): “Most work applied pre-trained speech models to only inference problems such as ASR (Baevski et al., 2019) or phone discrimination (Kharitonov et al., 2020), with a notable exception being Tjandra et al. (2019a), which focuses on text-free machine translation.”
• Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation (2022, unknown): “S2ST, which directly translates the source speech to the target speech, has attracted a lot of attention recently [1,2,20–22].”
• Translatotron 2: Robust direct speech-to-speech translation (2021, background): “The main contributions include: Tjandra et al. (2019) introduced an S2ST system that first translates the source speech into a discrete representation of the target speech which is predicted from a separately trained VQ-VAE (Oord et al., 2017), and then uses the VQ-VAE decoder to convert the…”
• Translatotron 2: High-quality direct speech-to-speech translation with voice preservation (2021, methodology): “, 2021) and cascade S2ST based on discrete speech representation (Tjandra et al., 2019; Zhang et al., 2021; Lee et al., 2022; 2021a; Ma et al., 2021).”