Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation

By Sravya Popuri, Peng-Jen Chen, Changhan Wang, Juan Pino, Yossi Adi, Jiatao Gu, Wei-Ning Hsu, Ann Lee. 2022

70 citations recorded by the source corpus.

Direct speech-to-speech translation (S2ST) models suffer from data scarcity issues as there exists little parallel S2ST data, compared to the amount of data available for conventional cascaded systems that consist of automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS) synthesis. In this work, we explore self-supervised pre-training with unlabeled speech data and data augmentation to tackle this issue. We take advantage of a recently proposed speech-to-unit translation (S2UT) framework that encodes target speech into discrete representations, and transfer pre-training and efficient partial finetuning techniques that work well for speech-to-text translation (S2T) to the S2UT domain by studying both speech encoder and discrete unit decoder pre-training. Our experiments on Spanish-English translation show that self-supervised pre-training consistently improves model performance compared with multitask learning with an average 6.6-12.1 BLEU gain, and it can be further combined with data augmentation techniques that apply MT to create weakly supervised training data. Audio samples are available at: https://facebookresearch.github.io/speech_translation/enhanced_direct_s2st_units/index.html .

Subfields: Artificial Intelligence, Machine Translation, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yossi Adi
2. Gabriel Synnaeve
3. Wei-Ning Hsu
4. Juan Pino
5. Alexis Conneau
6. Changhan Wang
7. Mohit Bansal
8. Ann Lee

In Machine Translation

1. Juan Pino
2. Changhan Wang
3. Alexis Conneau
4. Ye Jia
5. Yossi Adi
6. Ann Lee
7. Tal Remez
8. Holger Schwenk

In Speech & Audio

1. Yossi Adi
2. Gabriel Synnaeve
3. Wei-Ning Hsu
4. Jade Copet
5. Tal Remez
6. Juan Pino
7. Changhan Wang
8. Alexis Conneau

Citations in the corpus

• High Fidelity Neural Audio Compression (2022, methodology): “Similar methods were applied to speech resynthesis (Polyak et al., 2021), speech emotion conversion (Kreuk et al., 2021), spoken dialog system (Nguyen et al., 2022), and speech-to-speech translation (Lee et al., 2021a;b; Popuri et al., 2022).”
• AudioGen: Textually Guided Audio Generation (2022, background): “…modeling (Lakho-2 tia et al., 2021; Kharitonov et al., 2022a; Borsos et al., 2022), multi-stream processing (Kharitonov et al., 2022b), speech emotion conversion (Kreuk et al., 2022), spoken dialuge (Nguyen et al., 2022), and speech-to-speech translation (Lee et al., 2022a;b; Popuri et al., 2022).”
• SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2022, methodology): “Inspired by the recent finding that crossmodal pre-training is beneficial for speech translation (Popuri et al., 2022), we apply XM Transformer to multilingual training, whose encoder is initialized from pre-trained XLS-”
• Textually Pretrained Speech Language Models (2023, methodology): “Lee et al. [2021, 2022], Popuri et al. [2022] followed a similar modeling mechanism using a different speech tokenizer and proposed a textless approach for speech-to-speech translation.”
• Textless Direct Speech-to-Speech Translation with Discrete Speech Representation (2022, background): “Besides discrete speech representation, continuous speech representations learned on unsupervised data [21–25] have also been shown effective in S2ST [10, 19].”
• Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation (2022, unknown): “The first one employs HuBERT encoder to initialize the encoder of speech-to-unit translation model, and the other is existing S2UT model [3], which is initialized with HuBERT encoder plus 6-layer unit level mBART decoder.”