Janus-III: speech-to-speech translation in multiple languages

By Alon Lavie, A. Waibel, Lori S. Levin, Michael Finke, D. Gates, Marsal Gavaldà, T. Zeppenfeld, P. Zhan. 1997

Subfields: Machine Translation, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Machine Translation

1. Ye Jia
2. Juan Pino
3. Changhan Wang
4. Yossi Adi
5. Yun Tang
6. Xutai Ma
7. Ann Lee
8. Michelle Tadmor Ramanovich

In Speech & Audio

1. Xu Tan
2. Tao Qin
3. Yi Ren
4. Zhou Zhao
5. Wei-Ning Hsu
6. Tie-Yan Liu
7. Ye Jia
8. Yossi Adi

Citations in the corpus

• AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “Conventional automatic speech-to-speech translation systems are typically composed of a cascade of three components: automatic speech recognition (ASR), text-to-text machine translation (MT), and text-to-speech (TTS) synthesis [Lavie et al., 1997, Wahlster, 2000, Nakamura et al., 2006].”
• PolyVoice: Language Models for Speech to Speech Translation (2023, methodology): “Speech-to-speech translation (Lavie et al., 1997; Baldridge, 2004; Nakamura et al., 2006) aims to develop models capable of generating target language speech from source language speech.”
• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, neutral): “The initial research and application mainly focus on cascaded S2ST systems [Lavie et al., 1997, Nakamura et al., 2006, Wahlster, 2013], consisting of speech recognition (ASR), machine translation (MT), and speech synthesis (TTS) models.”
• Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, background): “Compared to conventional cascaded approaches [7, 8], which take advantage of automatic speech recognition (ASR), machine translation (MT) or end-to-end speech-to-text translation (S2T) followed by text-to-speech synthesis (TTS), direct S2ST has the advantage of faster inference [4] and can support t”
• Textless Speech-to-Speech Translation on Real Data (2021, neutral): “Conventional S2ST systems (Lavie et al., 1997; Nakamura et al., 2006) usually rely on a cascaded approach by first translating speech into text in the target language, either with automatic speech recognition (ASR) followed by machine tranlsation (MT), or an end-to-end speech-to-text translation…”
• Direct Speech-to-Speech Translation With Discrete Units (2021, methodology): “Conventional S2ST systems are built by combining either cascaded or end-to-end S2T models with TTS [13, 14].”
• Direct speech-to-speech translation with a sequence-to-sequence model (2019, neutral): “This is in contrast to conventional S2ST systems which are often broken down into three components: automatic speech recognition (ASR), text-to-text machine translation (MT), and text-to-speech (TTS) synthesis [1–4].”
• UWSpeech: Speech to Speech Translation for Unwritten Languages (2020, background): “Speech to speech translation (Lavie et al. 1997; Nakamura et al. 2006; Wahlster 2013; Jia et al. 2019) is important to help the understanding of cross-lingual spoken conversations and lectures, and has been used in scenarios such as international travel or conference.”
• Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translation (2022, methodology): “The conventional S2ST is implemented by a cascaded system [1], which includes three components: automatic speech recognition (ASR), text-to-text machine translation (MT), and textto-speech synthesis (TTS).”
• Translatotron 2: Robust direct speech-to-speech translation (2021, background): “Conventional S2ST systems are often composed of a cascade of three components: automatic speech recognition (ASR), text-to-text machine translation (MT), and text-to-speech (TTS) synthesis (Lavie et al., 1997; Wahlster, 2000; Nakamura et al., 2006).”
• Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation (2022, unknown): “Conventional speech to speech translation is usually composed of cascaded ASR, MT and TTS modules [16, 17].”
• TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation (2022, methodology): “Among the conventional method (Lavie et al., 1997; Nakamura et al., 2006; Wahlster, 2013), the cascaded system of automatic speech recognition (ASR), machine translation (MT), or speech-to-text translation (S2T) followed by text-to-speech synthesis (TTS) have demonstrated reasonable results yet…”
• Translatotron 2: High-quality direct speech-to-speech translation with voice preservation (2021, methodology): “Related works S2ST Until very recently, automatic S2ST systems are typically composed of a cascade of ASR, MT, and TTS components (Lavie et al., 1997; Wahlster, 2000; Nakamura et al., 2006; ITU, 2016).”