Direct speech-to-speech translation with a sequence-to-sequence model
By Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Z. Chen, Yonghui Wu. 2019
281 citations recorded by the source corpus.
We present an attention-based sequence-to-sequence neural network which can directly translate speech from one language into speech in another language, without relying on an intermediate text representation. The network is trained end-to-end, learning to map speech spectrograms into target spectrograms in another language, corresponding to the translated content (in a different canonical voice). We further demonstrate the ability to synthesize translated speech using the voice of the source speaker. We conduct experiments on two Spanish-to-English speech translation datasets, and find that the proposed model slightly underperforms a baseline cascade of a direct speech-to-text translation model and a text-to-speech synthesis model, demonstrating the feasibility of the approach on this very challenging task.
Subfields: Artificial Intelligence, Natural Language Processing, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Natural Language Processing
In Machine Translation
Citations in the corpus
- WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “Deep generative models have revolutionized speech synthesis (Oord et al., 2016; Sotelo et al., 2017; Wang et al., 2017; Biadsy et al., 2019; Jia et al., 2019).”
- VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, background): “Finally, there has been little research (Jia et al., 2019; Tjandra et al., 2019; Zhang et al., 2020a) into speech output due to the lack of open data.”
- AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “Direct speech-to-speech translation systems [Jia et al., 2019b, Kano et al., 2021, Jia et al., 2022b] are trained end-to-end operating on the audio spectrogram domain without relying on text representation at inference time.”
- PolyVoice: Language Models for Speech to Speech Translation (2023, background): “Recently, end-to-end paradigms (Jia et al., 2019) have gained popularity in the field of S2ST, as they allow for a single model to perform one or more of the afore-mentioned tasks, which consequently reduces error propagation and latency.”
- Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, positive): “At the same time, Translatotron [Jia et al., 2019] tries to synthesize target speech conditioned by the speaker embedding extracted from the source speech, but it misses richer voice information due to the limitation of the speaker embedding.”
- Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, background): “For direct S2ST, [1, 4] show that multitask learning is crucial for model convergence, and [1, 19] focus on incorporating pretrained modules from ASR, S2T, MT or TTS tasks.”
- Textless Speech-to-Speech Translation on Real Data (2021, neutral): “…a speech retrieval task (Zanon Boito et al., 2020) or human evaluation of the data quality (Duquenne et al., 2021), since existing direct S2ST systems are mostly trained with synthetic target speech (Jia et al., 2019; Tjan-dra et al., 2019; Zhang et al., 2020; Lee et al., 2021; Jia et al., 2021).”
- BigVGAN: A Universal Neural Vocoder with Large-Scale Training (2022, background): “speakers [39], neural voice cloning [2, 13], voice conversion [26], speech-to-speech translation [12], and neural audio codec [59], where the studio-quality recordings are often not available.”
- UWSpeech: Speech to Speech Translation for Unwritten Languages (2020, methodology): “Previous works on speech translations has evolved from cascaded models [19, 24, 26, 27, 42] to end-to-end models [5, 15, 33, 41, 43], where the text corresponding to speech are leveraged as auxiliary training for better accuracy.”
- Speech-to-Speech Translation Between Untranscribed Unknown Languages (2019, background): “[8] also observed a similar result with a similar scenario.”
- Europarl-ST: A Multilingual Corpus for Speech Translation of Parliamentary Debates (2019, background): “Moreover, recent studies report their main results using either the paid Fisher/Callhome corpora [1, 4, 5, 6, 10], or private propietary datasets [8], which limits reproducibility for the research community.”
- CVSS Corpus and Massively Multilingual Speech-to-Speech Translation (2022, positive): “On each version of CVSS, we trained two baseline end-to-end direct S2ST models following Translatotron (Jia et al., 2019b) and Translatotron 2 (Jia et al., 2022).”
- UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation (2021, methodology): “Vocoders have been employed in various fields such as text-tospeech[1, 2, 3, 4], voice conversion[5], and speech-to-speech translation[6].”
- A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, background): “Translatotron [167] translated from voice input to voice output through a sequential process.”
- Translatotron 2: Robust direct speech-to-speech translation (2021, methodology): “Very recently, direct speech-to-text translation (ST) is rapidly emerging, and has outperformed the cascade of ASR and MT (Weiss et al., 2017; Jia et al., 2019a; Di Gangi et al., 2019; McCarthy et al., 2020; Wang et al., 2021), which makes the cascade of ST and TTS as S2ST feasible (Jia et al.,…”
- Translatotron 2: High-quality direct speech-to-speech translation with voice preservation (2021, methodology): “While most of these works conducted experiments using synthetic datasets with translation speech in a clean single speaker’s voice, Jia et al. (2019b); Lee et al. (2021a) conducted experiments using multi-speaker human recordings.”