Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation
By Ye Jia, Yifan Ding, Ankur Bapna, Colin Cherry, Yu Zhang, Alexis Conneau, Nobuyuki Morioka. 2022
25 citations recorded by the source corpus.
End-to-end speech-to-speech translation (S2ST) without relying on intermediate text representations is a rapidly emerging frontier of research. Recent works have demonstrated that the performance of such direct S2ST systems is approaching that of conventional cascade S2ST when trained on comparable datasets. However, in practice, the performance of direct S2ST is bounded by the availability of paired S2ST training data. In this work, we explore multiple approaches for leveraging much more widely available unsupervised and weakly-supervised speech and text data to improve the performance of direct S2ST based on Translatotron 2. With our most effective approaches, the average translation quality of direct S2ST on 21 language pairs on the CVSS-C corpus is improved by +13.6 BLEU (or +113% relatively), as compared to the previous state-of-the-art trained without additional data. The improvements on low-resource language are even more significant (+398% relatively on average). Our comparative studies suggest future research directions for S2ST and speech representation learning.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Juan Pino
- Changhan Wang
- Ye Jia
- Hongyu Gong
- Ann Lee
- Paul-Ambroise Duquenne
- Holger Schwenk
- Chung-Cheng Chiu
Citations in the corpus
- Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation (2022, unknown): “For example, the paper in [9] explores pre-training the encoder with mSLAM objective [10], and pre-training the decoder of Translatoron 2 [11] with MT task to generate phonemes.”
- Textless Direct Speech-to-Speech Translation with Discrete Speech Representation (2022, methodology): “Besides discrete speech representation, continuous speech representations learned on unsupervised data [21–25] have also been shown effective in S2ST [10, 19].”
- SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2022, background): “Translatotron (Jia et al., 2019) and Translatotron2 (Jia et al., 2022b) train end-to-end S2ST to generate target spectrograms with multitask learning.”