Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translation

By Qianqian Dong, Fengpeng Yue, Tom Ko, Mingxuan Wang, Qibing Bai, Yu Zhang. 2022

19 citations recorded by the source corpus.

Direct Speech-to-speech translation (S2ST) has drawn more and more attention recently. The task is very challenging due to data scarcity and complex speech-to-speech mapping. In this paper, we report our recent achievements in S2ST. Firstly, we build a S2ST Transformer baseline which outperforms the original Translatotron. Secondly, we utilize the external data by pseudo-labeling and obtain a new state-of-the-art result on the Fisher English-to-Spanish test set. Indeed, we exploit the pseudo data with a combination of popular techniques which are not trivial when applied to S2ST. Moreover, we evaluate our approach on both syntactically similar (Spanish-English) and distant (English-Chinese) language pairs. Our implementation is available at https://github.com/fengpeng-yue/speech-to-speech-translation.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiying Huang
2. Qiao Tian
3. Fengpeng Yue
4. Yuxuan Wang
5. Yuping Wang
6. Zejun Ma
7. Tom Ko
8. Qianqian Dong

Citations in the corpus

• PolyVoice: Language Models for Speech to Speech Translation (2023, background): “Existing direct S2ST approaches can be further classified according to whether the model predicts continuous mel-spectrogram features (Dong et al., 2022) or discrete units (Lee et al., 2022).”
• Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation (2022, unknown): “To alleviate this problem, a line of work tries to leverage pseudo data to improve direct S2ST [3, 8].”