Translatotron 2: Robust direct speech-to-speech translation

By Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, Roi Pomerantz. 2021

31 citations recorded by the source corpus.

Subfields: Machine Translation, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Machine Translation

1. Ye Jia
2. Juan Pino
3. Changhan Wang
4. Yun Tang
5. Yossi Adi
6. Xutai Ma
7. Ann Lee
8. Alexis Conneau

In Speech & Audio

1. Ye Jia
2. Wei-Ning Hsu
3. Yossi Adi
4. Yu Zhang
5. Yi Ren
6. Zhou Zhao
7. Juan Pino
8. Changhan Wang

Citations in the corpus

• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, neutral): “Translatotron 2 [Jia et al., 2021] retrains the speaker voices relying on the pseudo bilingual speech data of the same speakers generated by multi-speaker TTS systems, while the synthetic speech does not completely simulate the speech of the real world.”
• Textless Speech-to-Speech Translation on Real Data (2021, neutral): “, 2021) or for learning a decoder that generates intermediate representations (Jia et al., 2021; Kano et al., 2021) to”
• Direct Speech-to-Speech Translation With Discrete Units (2021, methodology): “The follow-up and concurrent work, Translatotron 2 (Jia et al., 2021), addresses the over-generation issue by conditioning the spectrogram synthesizer directly on the output from the auxiliary target phoneme decoder.”
• CVSS Corpus and Massively Multilingual Speech-to-Speech Translation (2022, positive): “It synthesizes speech as natural as professional human speakers (Jia et al., 2021).”
• TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation (2022, methodology): “Translatotron 2 [6] utilizes the auxiliary target phoneme decoder to promote translation quality but still needs phoneme data during training.”
• Direct simultaneous speech to speech translation (2021, background): “Translatron2 is further proposed with improvements in voice preservation, speech naturalness and prediction robustness over Transla-tron (Jia et al., 2021).”
• Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation (2022, methodology): “speakers’ voices) [1, 3], supporting languages without written form [4–6, 8], avoiding error compounding across sub-systems and better handling of content that does not need to be translated (e.”
• Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translation (2022, methodology): “After that, [4] proposes Translatotron2 to improve the robustness of the predicted speech and retain the source speaker’s voice in the translated speech better.”
• Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation (2022, unknown): “For example, the paper in [9] explores pre-training the encoder with mSLAM objective [10], and pre-training the decoder of Translatoron 2 [11] with MT task to generate phonemes.”