Transformer-Based Direct Speech-To-Speech Translation with Transcoder

By Takatomo Kano, Sakriani Watiasri Sakti, Satoshi Nakamura. 2021

62 citations recorded by the source corpus.

Traditional speech translation systems use a cascade manner that concatenates speech recognition (ASR), machine translation (MT), and text-to-speech (TTS) synthesis to translate speech from one language to another language in a step-by-step manner. Unfortunately, since those components are trained separately, MT often struggles to handle ASR errors, resulting in unnatural translation results. Recently, one work attempted to construct direct speech translation in a single model. The model used a multi-task scheme that learns to predict not only the target speech spectrograms directly but also the source and target phoneme transcription as auxiliary tasks. However, that work was only evaluated Spanish-English language pairs with similar syntax and word order. With syntactically distant language pairs, speech translation requires distant word order, and thus direct speech frame-to-frame alignments become difficult. Another direction was to construct a single deep-learning framework while keeping the step-by-step translation process. However, such studies focused only on speech-to-text translation. Furthermore, all of these works were based on a recurrent neural net-work (RNN) model. In this work, we propose a step-by-step scheme to a complete end-to-end speech-to-speech translation and propose a Transformer-based speech translation using Transcoder. We compare our proposed and multi-task model using syntactically similar and distant language pairs.

Subfields: Artificial Intelligence, Model Architectures, Machine Translation, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Machine Translation

In Speech & Audio

Citations in the corpus

AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “Direct speech-to-speech translation systems [Jia et al., 2019b, Kano et al., 2021, Jia et al., 2022b] are trained end-to-end operating on the audio spectrogram domain without relying on text representation at inference time.”
PolyVoice: Language Models for Speech to Speech Translation (2023, background): “Among the various techniques, auxiliary supervision based on textual data has been particularly effective during training (Jia et al., 2019; Kano et al., 2021).”
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, background): “For direct S2ST, [1, 4] show that multitask learning is crucial for model convergence, and [1, 19] focus on incorporating pretrained modules from ASR, S2T, MT or TTS tasks.”
CVSS Corpus and Massively Multilingual Speech-to-Speech Translation (2022, neutral): “Synthetic vs Human-recorded Most of the above mentioned recent S2ST works use synthetic translation speech as training targets (Jia et al., 2019b; Tjandra et al., 2019; Zhang et al., 2021; Kano et al., 2021; Lee et al., 2021a; Jia et al., 2021a; Ma et al., 2021).”
Translatotron 2: High-quality direct speech-to-speech translation with voice preservation (2021, methodology): “Kano et al. (2021) introduced an end-to-end S2ST model with a cascade of three autoregressive decoders, and used pre-trained MT and TTS models as teacher models to facilitate the training of the end-to-end model.”
Translatotron 2: Robust direct speech-to-speech translation (2021, methodology): “Kano et al. (2021) introduced an end-to-end S2ST model with a cascade of three auto-regressive decoders, and used pre-trained MT and TTS models as teacher models to facilitate the training of the end-to-end model.”