Translatotron 2: High-quality direct speech-to-speech translation with voice preservation

By Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, Roi Pomerantz. 2021

118 citations recorded by the source corpus.

We present Translatotron 2, a neural direct speech-to-speech translation model that can be trained end-to-end. Translatotron 2 consists of a speech encoder, a linguistic decoder, an acoustic synthesizer, and a single attention module that connects them together. Experimental results on three datasets consistently show that Translatotron 2 outperforms the original Translatotron by a large margin on both translation quality (up to +15.5 BLEU) and speech generation quality, and approaches the same of cascade systems. In addition, we propose a simple method for preserving speakers' voices from the source speech to the translation speech in a different language. Unlike existing approaches, the proposed method is able to preserve each speaker's voice on speaker turns without requiring for speaker segmentation. Furthermore, compared to existing approaches, it better preserves speaker's privacy and mitigates potential misuse of voice cloning for creating spoofing audio artifacts.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

AudioPaLM: A Large Language Model That Can Speak and Listen (2023, positive): “Direct speech-to-speech translation systems [Jia et al., 2019b, Kano et al., 2021, Jia et al., 2022b] are trained end-to-end operating on the audio spectrogram domain without relying on text representation at inference time.”
PolyVoice: Language Models for Speech to Speech Translation (2023, methodology): “We evaluate our method on two speech-to-speech benchmark datasets, EMIME (Wester and Liang, 2011) and CVSS (Jia et al., 2022b).”
Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, neutral): “Since previous work shows that current end-to-end S2ST systems underperform cascaded S2ST systems [Jia et al., 2022a, Lee et al., 2021b], we also build an S2ST baseline which is cascaded by an ASR model, an MT model, and a multi-speaker YourTTS.”
CVSS Corpus and Massively Multilingual Speech-to-Speech Translation (2022, methodology): “Each S2ST pair has similar voices on the two sides despite being in different languages, making this dataset suitable for building models that preserve speakers’ voices when translating speech into different languages (Jia et al., 2021a).”
Textless Direct Speech-to-Speech Translation with Discrete Speech Representation (2022, methodology): “In this work, we propose a novel model, Textless Translatotron, which is based on Translatotron 2 [1], for training an end-to-end direct S2ST model without any textual supervision.”
Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation (2022, methodology): “speakers’ voices) [1, 3], supporting languages without written form [4–6, 8], avoiding error compounding across sub-systems and better handling of content that does not need to be translated (e.”
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2022, background): “Translatotron (Jia et al., 2019) and Translatotron2 (Jia et al., 2022b) train end-to-end S2ST to generate target spectrograms with multitask learning.”